logo

Manual Pages


Table of Contents

NAME

na_nfsstat - display NFS statistics

SYNOPSIS

nfsstat interval [ count ]

nfsstat -h [ -n [ count ] ] | [ [ ip_address | host_name ] [ interval ] ]

nfsstat -l [ count ]

nfsstat -z

nfsstat -d

nfsstat -C

nfsstat [ -c ] [ -t ]

DESCRIPTION

nfsstat displays statistical information about NFS (Network File System) and RPC (Remote Procedure Call) for the filer. That includes statistics for NFS versions 2, 3 and 4. It can also be used to reinitialize this information. If no arguments are given, nfsstat displays statistical information since last zeroed with the -z option (or since reboot if statistics have not been zeroed).

If just the interval argument is specified, nfsstat continuously displays the summary information for the following NFS requests: getattr, lookup, readlink, read, write, create, remove, and readdir/readdirplus. The output will show incremental statistics for the interval (in seconds) since the command was executed. If count argument is also specified, then the command exits after displaying the statistics for the specified number of intervals. The interval specified must be an integer.

Per-client statistics can also be collected and displayed by enabling the nfs.per_client_stats.enable options (using the options command - see na_options(1)) and invoking nfsstat with the -h or the -l options. Per-client statistics can be collected for up to the first 1000 NFS clients, that have mounted the filesystem on the given filer. These statistics are displayed in the decreasing order of the total NFS ops performed by each client.

OPTIONS

-h
Displays per-client statistics since last zeroed with the -z option (or since reboot if statistics have not been zeroed). The statistics are displayed on a per-client basis, with the IP address and host name (where available) of each client being displayed at the head of each client's block of statistics.

If optional count is specified using -n option, then per-client statistics for that many clients are displayed.

If an optional IP address or host name is specified with the -h option, only the statistics associated with this client are displayed.

Statistics for read and write sizes are also displayed as number of read or write requests made by each client grouped by size of request. Without this option, the cumulative statistics for all clients are displayed.

If the IP address or host name is specified, an optional interval argument can be supplied. This will cause interval statistics to be displayed for the specified client. The behavior is identical as described in the Description paragraph.

-l
Displays a list of the clients whose statistics have been collected on a per-client basis, along with the total NFS calls for that client since last reboot, or last zeroed with the -z option, or last explicit user request to enable the per-client statistics, the count being displayed both as the actual count and as a percentage of calls from all clients. The list is sorted in the decreasing order of the total calls for each client. If count argument is specified, then the command exits after displaying the statistics for the first count number of clients and if it is not specified the statistics for up to the first 256 clients are displayed, ordered by the total NFS ops performed by each client.

-z
Zeroes (reinitializes) the current cumulative and per client statistics. (However, statistics since boot are retained.) The per clients statistics also get reinitialized when nfs.per_client_stats.enable option is turned off.

-c
Includes reply cache statistics in the data displayed. The reply cache is used to cache replies to non-idempotent nfs requests, so that in case of a retransmit, the correct reply can be sent back to the client. Non-idempotent operations do not return same results if repeated e.g. remove, create. The various fields displayed are:

In Progress
Number of requests which were being processed at the time retransmit requests ware received for them.

Delay Hits
Number of requests which were dropped because they came in within nfs_rc_delay of sending out the reply. This field is deprecated. A zero value will always be printed for this field. This is to maintain backward compatibility with previous ONTAP releases.

Misses
Number of requests that were not found in the reply cache. These also include requests coming in for the first time.

Idempotent
Number of idempotent requests saved in the reply cache for the purpose of improving performance.

Non-idempotent Number of non-idempotent requests replied to from the reply cache.

-t
Displays the statistics since boot time, rather than since the last time they were zeroed.

-C
Displays the number and type of NFS version 2 and 3 requests received by all FlexCache volumes. In addition we display the number of each type that resulted in a cache hit.

-d
In addition to reply cache statistics, also includes statistics about incoming messages and allocated mbufs. The various fields displayed are:

nfs cache size
NFS reply cache size. The cache size depends on the Filer memory.

hash size
NFS reply cache hash size.

num msg
Number of incoming requests (over UDP) dropped because there are no free entries in the NFS operation table.

too many mbufs
Number of requests (over UDP) dropped because they require more mbufs than are available.

rpcErr
Total number of rpc errors encountered while generating the rpc reply header.

svrErr
Total number of RPC errors encountered while processing the request.

msg queued
Total number of NFS connections (over TCP) that have been deferred waiting for NFS resources to become available. The NFS requests queued on these connections will be processed once these resources become available. Connections can get queued up when the filesystem is busy doing back to back CPs.

no msg re-queued(xfc)
Total number of NFS connections (over TCP) which are in transmit flow control and for which further request processing has been deferred because there are too many replies queued on the socket waiting to be acknowledged by the client.

no msg dropped
Number of incoming requests (over UDP) dropped because there are no free entries available in the NFS operation table. Its the same as num msg. Requests can be dropped when the filesystem is busy doing back to back CPs.

no msg unqueued
Number of connections unqueued from the deferred list.

no msg discarded
Number of deferred connections, waiting for NFS resources to become available, dropped because of a timeout or shutdown of the socket.

no msg dropped from vol offline Number of requests dropped because the volume was offline.

no deferred msg processed
Number of NFS requests unqueued and processed from NFS over TCP connections that had been put on a queue waiting for NFS resources to become available. This is not the same as number of NFS connections queued up waiting for resources. Note that one connection can contain more than one NFS requests waiting to be processed.

sbfull queued
Number of requests queued because the send buffer is full.

sbfull unqueued
Number of requests unqueued when space available in the send buffer.

sbfull discarded
Number of requests waiting on a full send buffer dropped because of a forced shutdown of the socket.

no mbuf queued
Number of requests (over TCP) waiting for mbufs to send a reply.

no mbuf dropped
Number of requests (over UDP) dropped because of unavailability of enough mbufs to send a reply.

no mbuf unqueued
Number of requests (over TCP) processed from the wait queue when enough mbufs become available.

no mbuf discarded Number of requests (over TCP) waiting for mbufs dropped because nothing can be sent over the socket. This can happen if the socket is closing.

(cumulative) active Two values are printed here. The maximum (cumulative) number of entries available in the NFS operation table and the total number of entries processing requests at that time.

req mbufs
Number of mbufs allocated for an incoming NFS request over UDP.

tcp input flowcontrol receive
Number of times the Filer went into receive flow control.

tcp input flowcontrol xmit
Number of times the Filer went into transmit flow control.

tcp input flowcontrol out receive Number of times the Filer came out of receive flow control.

tcp input flowcontrol out xmit
Number of times the Filer came out of transmit flow control.

sockets zapped nfs Number of nfs sockets disconnected because the send upcall argument in the underlying tcp socket had been zeroed.

sockets zapped tcp Number of tcp sockets disconnected because both the send and receive upcall arguments in that tcp socket had been zeroed.

no delegation
Number of times no delegation was handed out due to some error.

read delegation
Number of read delegations handed out to NFS version 4 clients.

write delegation
Number of write delegations handed out to NFS version 4 clients.

v4 acls set
Total number of acls set on files/dir through an explicit setattr command or when a file or dir is created and an acl is supplied in the initial attributes.

nfs msgs counts
The tot is the total number of available NFS messages. The unallocated is the number of messages which have yet to be allocated from the system. The free is the number of allocated messages which are not in use. The used is the number of allocated messages which are currently in use. The "VM cb heard" is how many times virtual memory pressure has asked for messages to be released back to the system. The "VM cb done" is how many messages were released back to the system.

nfs reply cache counts
The tot is the total number of available reply cache entries. The unallocated is the number of entries which have yet to be allocated from the system. The free is the number of allocated entries which are not in use. The used is the number of allocated entries which are currently in use. The "VM cb heard" is how many times virtual memory pressure has asked for entries to be released back to the system. The "VM cb done" is how many entries were released back to the system.

v2 mount (requested, granted, denied, resolving) The requested is the number of v2 mount requests. The granted is the number of mount requests granted access. The denied is the number denied access. The resolving is the number of mount requests currently awaiting access resolution.

v2 unmount (requested, granted, denied) The requested is the number of v2 unmount requests. The granted is the number of unmount requests granted. The denied is the number denied.

v2 unmount all (requested, granted, denied) The requested is the number of v2 unmount all requests. The granted is the number of unmount all requests granted. The denied is the number denied.

v3 mount (requested, granted, denied, resolving) The requested is the number of v3 mount requests. The granted is the number of mount requests granted access. The denied is the number denied access. The resolving is the number of mount requests currently awaiting access resolution.

v3 unmount (requested, granted, denied) The requested is the number of v3 unmount requests. The granted is the number of unmount requests granted. The denied is the number denied.

v3 unmount all (requested, granted, denied) The requested is the number of v3 unmount all requests. The granted is the number of unmount all requests granted. The denied is the number denied.

mount service requests (curr, total, max, redriven) These are the requests made to the mount service. These include requests for mount, umount, showmount, etc. The curr is the number of current requests to the mount service. The total is the total number of requests made to the mount service. The max is the high water mark of the curr counter. The redriven is the total number of mount service requests that needed to be redriven after an access cache population.

access cache (hits, partial misses, misses) A request coming into the cache can be looking for any combination of read, write and root attributes. The hits are the number of requests which were serviced completely by an entry already in the access cache. The partial misses are the number of requests that were not serviced completely by an entry in the access cache. The misses are the number of requests which were not serviced by the access cache because there was no entry in the cache for that client. Note that in general, the ratio hits to the sum of hits and partial misses + misses should be high. I.e., we want significantly more hits than misses.

access cache lookup requests (curr, total, max) The curr is the number of current requests which are being serviced. This number is also reprentative of the number of entries on the export worker queue. The total is the sum of all requests. The max is the high water mark of the curr counter.

access cache nodes(found, created) The found is the number of entries that were hit as a result of a cache lookup. The created is the total number of access cache nodes created since boot.

access cache requests(queued, unqueued) The queued is the number of requests that had to wait (and were queued) for the access cache to be populated before they could be processed. The unqueued is the number of requests that were unqueued and processed after the access cache was populated.

access cache requests unqueued by (flush, restore) These are the number of requests waiting for access cache population which were restarted when that access cache was flushed through the exportfs -f command or was restored from disk upon reboot or failover.

access cache read requests (queued, unqueued) The queued counter is for requests which got a cache miss for read access and had to wait because that information was not in the cache. The unqueued counter is for requests which were restarted once the read access information was inserted in the access cache.

access cache write requests (queued, unqueued) The queued counter is for requests which got a cache miss for write access and had to wait because that information was not in the cache. The unqueued counter is for requests which were restarted once the write access information was inserted in the access cache.

access cache root requests (queued, unqueued) The queued counter is for requests which got a cache miss for root access and had to wait because that information was not in the cache. The unqueued counter is for requests which were restarted once the root access information was inserted in the access cache.

access cache expired hits (total, read, write, root)
These are the number of hits in the access cache for read, write or root attribute where the access information for that attribute had not been refreshed for sometime. The attribute information expires based on the timeout value set in nfs.export.neg.timeout and nfs.export.pos.timeout.

access cache inserts (full, partial, dup, subnet, restore)
The full count represents the number of inserts into the access cache where all attributes upon which one or more requests were waiting, were resolved. As a result of a full insert one or more requests would be restarted. The partial count represents the number of inserts where some of the attributes upon which one or more requests were waiting, were resolved. No requests are restarted as a result of a partial insert. The dup count represents the number of times an insert was done for an attribute when the access cache already contained some information about that attribute. However its not neccessary that the same access information was inserted the second time around. e.g. If the cache contained information about node A.B.C.D as read being allowed and then another insert was done in the cache which changed this information to read being denied for A.B.C.D, this would be considered a dup insert. The subet count represents the number of times access information was inserted for a subnet. The restore count represents the number of times an insert was done based on information on disk. This happens upon reboot or cluster failover.

access cache refreshes requested (total, read, write, root)
These counts represent for each attribute the number of times stale information was returned by the cache and a refresh was requested for that attribute.

access cache refreshes done (total, read, write, root)
These counts represent for each attribute the number of times a refresh was done for that attribute.

access cache errors (query, insert, no mem) The query counter indicates the total number of errors (including no mem) encountered when doing a query in the cache. The insert counter indicates the total number of errors (including no mem) encountered when doing an insert in the cache. The no mem counter indicates the number of times a query or insert could not be done because memory was not available.

access cache nodes (flushed, harvested, harvests failed)
The flushed count represents the number of nodes removed from the access cache as a result of an exportfs -f command. The harvested count represents the number of nodes removed from the access cache which have not been accessed for more than nfs.export.harvest.timeout seconds. The harvests failed count represents the number of times a node was found in a state where requests were waiting for resolution on it and that node had not been accessed for more than nfs.export.harvest.timeout seconds. High value of this counter may point to issues with name resolution service.

access cache nodes (allocated, free) The allocated count represents the total number of nodes allocated by all the access caches. The free count represents the number of nodes which are not in use by any access cache.

access cache qctx (allocated, free) The allocated count represents the total number of query contexts allocated to queue requests waiting for resolution. The free count represents the number of contexts which are not in use.

access cache persistence errors(total) The total count represents the errors encountered while saving or restoring access cache data to/from disk.

access cache persistence nodes handled (restored,saved)
The restored count represents the number of nodes that were restored from the persistent access cache. The saved count represents the number of nodes saved since the last system boot. This is cumulative and periodically increases after periodic saving of access cache into persistent storage.

access cache persistence rules deleted (total) The total count represents the number of times we reference a rule which no longer exists while restoring content from persistent access cache. Contents for such rules cannot (and need not) be restored.

access cache persistence memchunks (allocated, freed)
The allocated count represents the number of memory chunks allocated for use while encoding access cache data for persistent storage. The freed represents the number of memory chunks freed (after having been allocated) while processing access cache data for persistent storage.

assist queue (queued, split mbufs, drop for EAGAIN) The queued is the number of NFS requests which are put on the assist queue. The "split mbufs" is the number of NFS requests in which data spans multiple mbufs. The "drop for EAGAIN" is the number of NFS requests which were put on the assist queue for servicing an access cache miss and for which name service errors resulted in the request being dropped.

NFS re-drive queue(curr, max, total) The curr is the number of NFS requests currently on the redrive queue, waiting to be processed. The max is the maximum number of NFS requests that were waiting on the re-drive queue at any given time. The total is the total number of NFS requests processed so far.

Direct NFS re-drive(memory, webNFS) Shows the number of NFS requests directly queued from the driver thread to the redrive queue. The memory is the number of the NFS requests queued up to the re-drive queue because of the failure of the non-blocking memory allocation. The webNFS is the number of webNFS requests put on the redrive queue.

Errors in the blocking export access check Shows the count of the errors received during blocking export access check.

RPCSEC_GSS context limit
The limit on the RPCSEC_GSS context table. This limit is defined via the nfs.rpcsec.ctx.high

option.
RPCSEC_GSS refers to the authentication protocol used by NFS (see RFC2203) for encapsulating Kerberos V5 authentication information. Therefore an RPCSEC_GSS context corresponds to a Kerberos V5 authentication session between the NFS client and server.

current context count
The number of RPCSEC_GSS contexts currently allocated.

maximum context count
The maximum number of RPCSEC_GSS contexts ever allocated since the last boot. Note that this value is not zeroed when the -z option is used.

context reclaim callbacks

context idle/expired scans

vm pressure callbacks
Each of these three statistics counts an asynchronous attempt (callback) to reclaim least recently used contexts from the RPCSEC_GSS context table.

The "context reclaim callbacks" statistic counts the number of callbacks triggered by the RPCSEC_GSS subsystem, due to a deferred reclaim (because ONTAP could not suspend).

The "context idle/expired scans" statistic counts the number of periodic scans to look for expired or idle contexts. A context is considered idle if it has not been used for the number of seconds set in the nfs.rpcsec.ctx.idle option.

The "vm pressure callbacks" statistic counts the number of times the ONTAP VM subsystem asked the RPCSEC_GSS subsystem to release memory back to the system free list of memory.

contexts created
Counts the total number of RPCSEC_GSS contexts created.

contexts deleted
Counts the total number of RPCSEC_GSS contexts deleted.

contexts deleted due to vm pressure Counts the RPCSEC_GSS contexts deleted due to vm pressure callbacks.

contexts deleted due to context limit Counts the RPCSEC_GSS contexts deleted due to the current context count already being at the context limit whenever a new context needs to be created. The least recently used context will be deleted.

contexts deleted due to idle/expiration Counts the number of RPCSEC_GSS contexts deleted because the expired or were idle.

requests exceeding timeout
The number of NFS requests which exceeded the timer set in the option nfs.response.trig_ger.

DISPLAYS

The server RPC display includes the following fields, with separate values for TCP and UDP:

calls
The total number of RPC calls received.

badcalls The total number of calls rejected by the RPC layer (the sum of badlen and xdrcall as defined below).

nullrecv The number of times an RPC call was not available when it was thought to be received.

badlen
The number of RPC calls with a length shorter than a minimum-sized RPC call. Also counts the number of RPC requests over the TCP transport with illegal record lengths, such as those that are too small, too large, or not a multiple of the base XDR encoding unit (4 bytes).

xdrcall The number of RPC calls whose header could not be XDR decoded.

The server NFS display shows the number of NFS calls received (calls) and rejected (badcalls), and the counts and percentages for the various calls that were made.

NFSV4 STATS

An NFSV4 compound request can contain more than one operations. Therefore, the nfsstat command displays the total number of rpc requests received by the server (calls) as well as the total operations (ops) received in all the compound requests.

Only a count is displayed for NULL and COMPOUND requests. Percentages are displayed for all operations received in COMPOUND requests relative to the total number of operations. The total calls displayed under the TCP, UDP and Server nfs headings represents the sum of all RPC requests received from NFSV2, NFSV3 and NFSV4 clients and do not take into account any operations in COMPOUND requests.

BUGS

nfsstat -l reports unexpected percentages, if the nfs.per_client_stats.enable option is enabled after the system has been running for a while (typically this option should be enabled at system startup time via the /etc/rc file). Resetting the statistics via the nfsstat -z command will clear this condition.

CLUSTER CONSIDERATIONS

In takeover mode, the nfsstat command displays combined statistics for the live filer and the failed filer. From the statistics, you cannot determine how many requests were serviced by the live filer and how many requests were serviced on behalf of the failed filer.

The -h and -l options display the combined client information for both the live filer and the failed filer.

The NFS statistics are cumulative. That is, a giveback operation does not zero out the NFS statistics. After giving back the failed filer's resources, the live filer does not subtract the statistics about NFS operations it performed on behalf of the failed filer when it was in takeover mode.

VFILER CONSIDERATIONS

When run from a vfiler context, (e.g. via the vfiler run command), nfsstat operates on the concerned vfiler. Note that if vfilers are licensed when using the -h and -l options it is necessary to run this command from a vfiler context, since the interpretations of the hosts are dependent on the vfiler.

SEE ALSO

na_netstat(1), na_options(1), na_partner(1), na_vfiler(1), na_sysstat(1).


Table of Contents