NFS Performance Analysis

NFS Performance Analysis

NFS collects statistics on the execution of various NFS operations, along with error information and performance indicators. You can use the following command to identify network problems and observe various NFS operations on your system.

Nfsstat command

: None. nfsstat command displays statistics about NFS and RPC interfaces to the client and server kernel. This command can also be used to reinitialize the counters for these statistics (nfsstat-Z ). Considering performance issues, RPC statistics (-r option) must be viewed first. The NFS statistics show you how the application uses NFS.

RPC statistics

: None. The nfsstat command displays statistics about RPC calls, for example:

• Total number of received or rejected RPC calls
• Total number of RPC calls sent from or rejected by the server
• Number of times the RPC package cannot be received while trying to receive it
• Number of packets with too short length or incomplete header information
• Number of times a call has to be resent
• Number of times the response and call do not match
• Number of times a call times out
• Number of times a call waits when the client handle is busy
• Number of times the authentication information has to be refreshed

: The NFS statistics output by the none. nfsstat command are divided into NFS V2 and NFS V3. The RPC statistics are dividedConnection-oriented(TCP) andNo connection(UDP.

NFS server information

For received NFS calls (CILS) And calls rejected because the authentication fails (Badcils) The NFS server displays the number of requests and the percentage of calls of many other types.

The following example shows the server output when the nfsstat command has the-S parameter option:

# nfsstat -s

Server rpc:
Connection oriented:
calls   badcalls  nullrecv badlen   xdrcall dupchecks dupreqs
15835      0          0          0          0          772        0
Connectionless:
calls    badcalls  nullrecv badlen   xdrcall dupchecks dupreqs
0          0          0          0          0          0          0

Server nfs:
calls      badcalls   public_v2  public_v3
15835      0          0          0
Version 2: (0 calls)
null       getattr    setattr    root       lookup     readlink   read
0 0%       0 0%       0 0%       0 0%       0 0%       0 0%       0 0%
wrcache    write      create     remove     rename     link       symlink
0 0%       0 0%       0 0%       0 0%       0 0%       0 0%       0 0%
mkdir      rmdir      readdir    statfs
0 0%       0 0%       0 0%       0 0%
Version 3: (15835 calls)
null       getattr    setattr    lookup     access     readlink   read
7 0%       3033 19%   55 0%      1008 6%    1542 9%    20 0%      9000 56%
write      create     mkdir      symlink    mknod      remove     rmdir
175 1%     185 1%     0 0%       0 0%       0 0%       120 0%     0 0%
rename     link       readdir    readdir+   fsstat     fsinfo     pathconf
87 0%      0 0%       1 0%       150 0%     348 2%     7 0%       0 0%
commit
97 0%

The RPC output server section (-S) includes the following items:

CILS

Total number of RPC calls received from the client

Badcils

Total number of calls rejected by the RPC Layer

Nullrecv

The number of times that an RPC call is considered to have been received but not captured.

Badlen

Packet truncation or package corruption (number of RPC calls whose length is less than the minimum RPC call standard length)

Xdrcall

Number of RPC calls whose header cannot be decoded by extended data (XDR)

Dupchecks

Number of RPC calls found in the cache for repeated requests

Dupreqs

Number of detected repeated RPC calls

The output results also show the number of calls and their respective percentages.

Repetitive checks are provided for operations that cannot return the same result in two executions. The typical example is the RM command. The first RM command may be successful, but if the response is lost, the client resends the command. We hope that such repeated requests will be successful. At this time, the duplicate cache will be queried. If a duplicate request is found, it will be the same (successful) the result is returned as the result of the second duplicate request, just like the result produced by the first request.

By viewing the call percentage of different types of operations (such as getattr (), read (), write (), and readdir (), you can decide which type of optimization to use. For example, if the getattr () call percentage is very high, the cache of the tuning attribute may be effective. If the write () call percentage is too high, disk and LVM tuning is important. If the read () call percentage is too high, more memory is used to buffer files to improve performance.

NFS client information

The NFS client displays the number of sent and rejected calls and the number of received client handles (Clgets) And the number of calls and their respective percentages.

The following example shows the client distribution output by the nfsstat command with the-C parameter option:

# nfsstat -c

Client rpc:
Connection oriented
calls    badcalls  badxids timeouts newcreds badverfs timers
0          0          0          0          0          0          0
nomem    cantconn interrupts
0          0          0
Connectionless
calls    badcalls retrans  badxids  timeouts newcreds badverfs
6553       0          0          0          0          0          0
timers   nomem     cantsend
0          0          0

Client nfs:
calls      badcalls   clgets     cltoomany
6541       0          0          0
Version 2: (6541 calls)
null       getattr    setattr    root       lookup     readlink   read
0 0%       590 9%     414 6%     0 0%       2308 35%   0 0%       0 0%
wrcache    write      create     remove     rename     link       symlink
0 0%       2482 37%   276 4%     277 4%     147 2%     0 0%       0 0%
mkdir      rmdir      readdir    statfs
6 0%       6 0%       30 0%      5 0%
Version 3: (0 calls)
null       getattr    setattr    lookup     access     readlink   read
0 0%       0 0%       0 0%       0 0%       0 0%       0 0%       0 0%
write      create     mkdir      symlink    mknod      remove     rmdir
0 0%       0 0%       0 0%       0 0%       0 0%       0 0%       0 0%
rename     link       readdir    readdir+   fsstat     fsinfo     pathconf
0 0%       0 0%       0 0%       0 0%       0 0%       0 0%       0 0%
commit
0 0%

The client (-C) output by RPC includes the following items:

CILS

Total number of calls from a client to NFS.

Badcils

Total number of calls rejected by the RPC layer.

Retrans

The number of times that a call times out due to waiting for the response from the server. This applies only to RPC calls in connectionless transmission.

Badxid

The number of times the server receives a response but does not have any unfinished call matching it. This means that the server may spend too much time responding.

Timeouts

The number of times that the call times out due to waiting for the server response.

Newcreds

The number of times the authentication information has to be refreshed.

Badverfs

Number of times the call failed due to incorrect Verifier in the response.

Timers

The call timeout value is longer than or equal to the specified minimum call timeout value.

Nomem

The number of calls that fail due to memory allocation failure.

Cantconn

The number of failed calls due to the inability to connect to the server.

Interrupts

The number of times a call is interrupted by a signal before completion.

Cantsend

The number of failed sending times because the client cannot be connected.

The output results also show the number of calls and their respective percentages.

Considering performance monitoring issues, the nfsstat-C command provides information to show whether UDP packets are discarded on the network. If a network cannot process a packet, it may be discarded. Discarded data packets can be caused by the following factors: the response time of network hardware, the response time of network software, or the CPU that exceeds the load on the server. The discarded package is not actually lost because an alternative request will be sent to it.

In the RPC statisticsRetransThe column shows the number of requests that have timed out and resending due to waiting for a response. This situation is related to discarding UDP packets. IfRetransWhen the number of calls in this item continuously exceeds 5% of the total number of calls in the first column, it indicates that there is a problem and the server can no longer meet the requirements. You can use the vmstat, netpmon, and iostat commands on the server to check the load.

Very largeBadxidThe number of requests means that the requests are sent to different NFS servers, but the server cannot respond before the client's RPC call times out and resends due to heavy load. When a sent request receives a duplicate responseBadxidThe value is incremented by one (each RPC Request retains its own Xid throughout the transmission life cycle ). Too many resend requests impose additional pressure on the server, further increasing the response time. IfBadxidAndTimeoutsThe number of calls exceeds 5% of the total number of calls. You can use the Smitty chnfsmnt command to increase the timeo parameter in the NFS-mount option. IfBadxidIs 0, is 0,RetransAndTimeoutsYou can try to reduce the NFS buffer size (use the rsize and wsize options in the mount command ).

If the server is bound with a CPU (CPU-bound), NFS and Its daemon will be affected. To improve performance in this case, the server must be optimized, upgraded, or the user can place application files on the local machine. If the server is bound with I/O (I/O-bound), the Server File System can be reorganized or local files can be used.

If the number of resends and timeouts is close to the same value, you can confirm that the packet is being discarded. Packets are rarely discarded on the client. Generally, a package is discarded either on the network or on the server. The server may discard packets in either of the following situations: when the packet overflows from the sending queue of the server interface driver, or when the server's User Datagram Protocol (UDP) when the socket buffer overflows (nfs_socketsize ). If it does not appear on the serverSocket buffer overflowOrOerrsIn addition, there are many resends and timeouts on the client, so the packet may be discarded on the network. The problem may be caused by a media or network device, such as a router, bridge, or hub. Network sniffer and other tools can be used for debugging and diagnosis. For more details, see discard package: none.

In some cases, the application or user has experienced poor performance. However, when the nfsstat-C command is used to check, the output results show that the number of times out and resends is equal to zero or few. This shows that the client receives responses from the server at the same speed as the request speed. First, check whether the number of biod daemon processes running on the client is appropriate. This check is still measurable when an application is locking a remote file. When a remote file lock is added to a file that is serving NFS, the client will disable all the data and attributes of the file in the cache and transfer it to a fully synchronous operation. As a result, execution performance is very slow. Unfortunately, this situation is quite common. Packet locking can be identified by looking for NLM requests from the output results of ipreport.

Nfsstat-m

: None. The nfsstat-M command displays the server name, server address, Mount flag, current read/write size, number of resends, and the number of dynamic resends of NFS mount on each client, as follows:

# nfsstat -m
/SAVE from /SAVE:itsorus.austin.ibm.com
 Flags:   vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5
 Lookups: srtt=27 (67ms), dev=17 (85ms), cur=11 (220ms)
 Reads:   srtt=16 (40ms), dev=7 (35ms), cur=5 (100ms)
 Writes:  srtt=42 (105ms), dev=14 (70ms), cur=12 (240ms)
 All:     srtt=27 (67ms), dev=17 (85ms), cur=11 (220ms)

The number in the parentheses in the output result of the preceding example is the actual time, in milliseconds. Other values are non-unit values maintained by the operating system kernel. You can ignore these unmeasured values. In the preceding example, the query, read, write, and all these operations are combined (All. Other definitions used in the output are as follows:

SRTT

Smoothed round-trip time)

Dev

Estimated Deviation)

Cur

Current Compensation timeout value (current backed-off timeout value)

Netpmon command

For more information about the command and its output, see the netpmon command.

Nfso command

: None. nfso command is used to configure the NFS attribute. You can set or display the network options of the currently running kernel. Therefore, this command can only be run after each system is started or configured.

Note: The none. nfso command does not perform a range check. If it is incorrectly used, the nfso command may make your system unable to work.

: None. nfso parameters and their respective values can be displayed using the nfso-a command, as shown below:

(tremor:) # nfso -a
portcheck= 0
udpchecksum= 1
nfs_socketsize= 60000
nfs_tcp_socketsize= 60000
nfs_setattr_error= 0
nfs_gather_threshold= 4096
nfs_repeat_messages= 0
nfs_udp_duplicate_cache_size= 5000
nfs_tcp_duplicate_cache_size= 5000
nfs_server_base_priority= 0
nfs_dynamic_retrans= 1
nfs_iopace_pages= 0
nfs_max_connections= 1024
nfs_max_threads= 128
nfs_use_reserved_ports= 0
nfs_device_specific_bufs= 1
nfs_server_clread= 1
nfs_rfc1323= 1
nfs_max_write_size= 0
nfs_max_read_size= 0
nfs_allow_all_signals= 0
nfs_v2_pdts= 1
nfs_v3_pdts= 1
nfs_v2_vm_bufs= 1000
nfs_v3_vm_bufs= 1000
nfs_securenfs_authtimeout= 0
nfs_v3_server_readdirplus= 1

For more information about these attributes, see network optimization parameters. Many NFS attributes are run-time attributes, which can be modified at any time. To modify the load time attribute (such as nfs_socketsize), you need to stop NFS before restarting.

To display or modify a specified parameter, run the nfso-O command as follows:

# nfso -o portcheck
portcheck= 0
# nfso -o portcheck=1

Parameters can also be reset to their default values. Use the-D option, as shown below:

# nfso -d portcheck
# nfso -o portcheck
portcheck= 0

NFS reference

The following is a summary of NFS-related files, commands, daemon processes, and subroutines. For more information, see AIX 5l v5.2 System Management Guide: communication and network and Aix 5l v5.2 command reference.

Network File System (NFS) file list

The following is a list of NFS files, including configuration information:

Bootparams

Lists the diskless clients that can be started.

Exports

Lists directories that can be exported to NFS clients.

Networks

Contains Network Information on the Internet

Pcnfsd. conf

Provides configuration options for the rpc. pcnfsd daemon

RPC

Contains the database information provided for Remote Procedure Call (RPC)

Xtab

Lists the exported directories.

/Etc/filesystems

Lists All file systems that are mounted upon system restart

NFS command list

The following is a list of NFS commands:

Chnfs

Start the specified number of biod and nfsd daemon

Mknfs

Configure the system to run NFS and start the NFS Daemon

Nfso

Configure NFS network options

Automount

Automatic mount NFS File System

Chnfsexp

Modify the attributes of the NFS-Exported Directory

Chnfsmnt

Modify the attributes of the NFS-mounted directory

Exportfs

Export directory to NFS client and recycle directory

Lsnfsexp

Displays the directory features exported by NFS

Lsnfsmnt

Show features of mounted NFS Systems

Mknfsexp

Use NFS to export a directory

Mknfsmnt

Mount a directory with NFS

Rmnfs

Stop the NFS Daemon

Rmnfsexp

Delete the NFS export (NFS-exported) directory from the server export column table

Rmnfsmnt

Delete the NFS mount (NFS-mounted) File System from the client Mount list

NFS Daemon list

The following is a list of NFC-locked daemon processes:

Lockd

Use RPC packets to process locked requests

STATD

Provides crash recovery (crash-and-recovery) for locked services on NFS.

The following is a list of network service daemon and utility tools:

Biod

Send client read/write requests to the server

Mountd

Respond to the Mount Request from the client to the file system

NFSD

Start the daemon process used to process client requests for file operations

Pcnfsd

Process service requests from PC-NFS clients

Nfsstat

Display the performance information of the machine receiving calls

On

Execute commands on a remote machine

Portmap

Map the RPC program port number to the Internet port number

Rexd

Receive requests from programs running on a remote machine

Rpcgen

Generate C code to implement an RPC protocol

Rpcinfo

Report the RPC server status

Rstatd

Returns the performance statistics obtained from the kernel.

RUP

Displays the status of a remote host on the local network.

Rusers

Report the list of users logged on to the remote machine

Rusersd

Response Request to the rusers command

Rwall

Send messages to all users on the network

Rwalld

Process the request of the rwall command

Showmount

Displays the list of all clients that have mounted remote file systems.

Spray

Sends a specified number of packets to a host

Sprayd

Receive packets sent from the spray command

The following is a list of Network Security daemon and utilities:

Chkey

Encryption key)

Keyenvoy

Provides mediation between user processes and key servers

Keylogin

Decrypt and store the user's secret key)

Keyserv

Store public and private keys

Mkkeyserv

Start the keyserv daemon and change the comments in the/etc/rc. NFS file to uncommented.

Newkey

Create a new key in the Public Key File

Rmkeyserv

Stop the keyserv daemon and comment out/etc/rc. Key set for The keyserv daemon in the NFS file

Ypupdated

Update information in the Network Information Service (NIS) Image

The following configuration files are supported by non-disk clients:

Bootparamd

Provide the information required to start a non-disk Client

The following is a list of NFS subprograms:

Cbc_crypt (), des_setparity (), or ecb_crypt ()

Implements the Data Encryption Standard (DES) routine.