iostat, vmstat, netstat – Performance Monitoring & Tuning in Unix & Linux

vmstat – Virtual Memory Statistics

vmstat reports virtual memory statistics of process, virtual memory, disk, trap, and CPU activity.

On multicpu systems , vmstat averages the number of CPUs into the output. For per-process statistics .Without options, vmstat displays a one-line summary of the virtual memory activity since the system was booted.

vmstat syntax

Basic synctax is vmstat interval count

option – let you specify the type of information needed such as paging -p , cache -c ,.interrupt -i etc.

if no option is specified information about process , memory , paging , disk ,interrupts & cpu is displayed .

interval – is time period in seconds between two samples . vmstat 4 will give data at each 4 seconds interval.

count – is the number of times the data is needed . vmstat 4 5 will give data at 4 seconds interval 5 times.

vmstat Example

The following command displays a summary of what the system
is doing every five seconds.

example% vmstat 5

     procs  memory          page             disk      faults        cpu
     r b w swap  free re mf pi p fr de sr s0 s1 s2 s3  in  sy  cs us sy id
     0 0 0 11456 4120 1  41 19 1  3  0  2  0  4  0  0  48 112 130  4 14 82
     0 0 1 10132 4280 0   4 44 0  0  0  0  0 23  0  0 211 230 144  3 35 62
     0 0 1 10132 4616 0   0 20 0  0  0  0  0 19  0  0 150 172 146  3 33 64
     0 0 1 10132 5292 0   0  9 0  0  0  0  0 21  0  0 165 105 130  1 21 78

The fields of vmstat's display are
procs
r     in run queue
b     blocked for resources I/O, paging etc.
w     swapped
memory (in Kbytes)
swap -  amount  of  swap   space   currently   available
free   - size of the free list

page ( in units per second).
re    page reclaims -  see  -S  option  for  how  this
field is modified.
mf    minor faults -  see  -S  option  for  how    this
field is modified.
pi    kilobytes paged in
po    kilobytes paged out
fr    kilobytes freed
de    anticipated short-term memory shortfall (Kbytes)
sr    pages scanned by clock algorithm
disk  ( operations per second )
There are  slots for up to four disks,
 labeled with a single letter and number. 
The letter indicates  the  type  of disk 
 (s = SCSI, i = IPI, etc). 
The number is  the logical unit number.

faults
in    (non clock) device interrupts
sy    system calls
cs    CPU context switches

cpu  -   breakdown of percentage usage of CPU  time.
 On multiprocessors  this is an a
 average across all processors.
us    user time
sy    system time
id    idle time

vmstat Results and Solution

A. CPU issues

Following columns has to be watched to determine if there is any cpu issue

1. Processes in the run queue (procs r)
2. User time (cpu us)
3. System time (cpu sy)
4. Idle time (cpu id)

     procs      cpu
     r b w    us sy  id
     0 0 0    4  14  82
     0 0 1    3  35  62
     0 0 1    3  33  64
     0 0 1    1  21  78

Problem symptoms

A.) Number of processes in run queue
1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs .
2.) if this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.
3.) If the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us) system is facing shortage of CPU resources.

Resolution
Resolution to these kind of issues involves tuning of application procedures to make efficient use of cpu and as a last resort increasing the cpu power or adding more cpu to the system.

B. Memory Issues

Memory bottlenecks are determined by the scan rate (sr) . The scan rate is the pages scanned by the clock algorithm per second. If the scan rate (sr) is continuously over 200 pages per second then there is a memory shortage.

Resolution

1. Tune the applications & servers to make efficient use of memory and cache.
2. Increase system memory .
3. Implement priority paging in s in pre solaris 8 versions by adding line “set priority paging=1” in
   /etc/system. Remove this line if upgrading from Solaris 7 to 8 & retaining old /etc/system file.

Next Page – netstat