Summary In this chapter, you learned how to back up and restore your file system. You learned how to choose which files are important to back up and to choose a backup medium, a backup method, and a tape rotation schedule appropriate for the needs of your situation. You also learned how to use low-level archiv- ing tools such as tar and dump to produce archives and file system data and to restore corrupted file system data from archives. In addition, you learned how to configure and use AMANDA, an advanced archiving tool. 804 Chapter 31 39_599496 ch31.qxd 8/30/05 7:13 PM Page 804 805 Performance Monitoring IN THIS CHAPTER ■■ System Performance Monitoring Tools ■■ Measuring Memory Usage ■■ Viewing Running Tasks ■■ Monitoring I/O Activity ■■ Using sar This chapter describes some of the tools you can use to monitor the status and performance of your Fedora Core or RHEL system. Utilities like free, top, and ps provide basic information about the status of the system at given points in time. For ongoing monitoring, you would use tools like iostat, vmstat, and sar. System-Performance-Monitoring Tools The first group of tools this chapter discusses enables you to take snapshots of system performance at a given point in time. You can use this data to create baseline metrics of your system’s performance. This historical data serves as a guide against which you measure the impact of changes you make. You can use a variety of tools, many more, in fact, than this chapter covers. The six you’ll look at are listed here in alphabetical order: ■■ free — Reports the amount of free and used memory in the system ■■ iostat — Provides detailed CPU and I/O usage information ■■ sar — Collects, saves, or reports on a comprehensive list of system activity data CHAPTER 32 40_599496 ch32.qxd 8/30/05 7:14 PM Page 805 ■■ slabtop — Reports kernel memory usage ■■ top — Displays a real-time list of running processes ■■ vmstat — Shows virtual memory and I/O system usage One of the things you will notice is that each utility has some overlap with other utilities. For example, free and vmstat both report on virtual memory usage, although vmstat provides considerably more detail than does free. Likewise, vmstat and iostat can both provide I/O (input/output) usage data; again, iostat’s I/O analysis is more complete than vmstat’s is. The following sections disregard these areas of overlap and focus on what each utility does best. For instance, you won’t see any discussion of vmstat’s disk I/O-specific features, nor will you read much about iostat’s ability to report on running processes (an area in which it overlaps with top). Measuring Memory Usage Even on systems that seem to have ample physical RAM, it is still a good idea to know how much memory is in use and how much is available. Excessive memory consumption, perhaps due to a memory leak in a running program, can slow a system down and eventually force a reboot to reclaim the “lost” memory. At the highest level, you can use free command to show a quick report of how much memory is in use and how much is free. vmstat shows more detail about memory usage, especially swap usage. The slabtop com- mand shows you how the kernel itself is allocating memory. Memory Usage as Seen by Users and Processes You can use two commands to obtain summary information about the system’s memory usage. The free command shows information about the amount of memory that is used and unused, including both physical RAM and swap space. vmstat shows the same information in greater detail. Free’s syntax is: free [-b|-k|-m] [-o] [-s secs] [-t] Invoked without command line arguments, free’s output looks like the following: $ free total used free shared buffers cached Mem: 515800 500652 15148 0 0 255356 -/+ buffers/cache: 245296 270504 Swap: 1052248 536 1051712 806 Chapter 32 40_599496 ch32.qxd 8/30/05 7:14 PM Page 806 If you want the output to be displayed in bytes, rather than kilobytes, use the -b option; use -m to display the output in megabytes; -k displays the out- put in kilobytes, the default. If you’re math challenged, the -t option adds a line to the bottom of the output showing totals values. The -o option disables the -/+ buffers/cached: line, which shows adjust- ments made to the used and free physical RAM. These adjustments are neces- sary if you want to know how much RAM is actually in use and how much RAM the kernel has set aside for its own use. The kernel keeps a certain amount of RAM available for I/O and memory buffers to facilitate I/O. The amount of buffer memory varies over time as it is used and released. From the point of view of the system as a whole, RAM used as buffer memory is always “in use,” even if the kernel has not allocated it at a given point in time. From the kernel’s point of view, however, unused buffer memory is just that, unused (or free). Without the -o option, you see memory usage from the kernel’s point of view. With the -o option, you can visualize memory consumption from the view of the system as a whole. In the free example just shown, just over 263 Mb (270,504 Kb) is “free” from the kernel’s point of view. If you use the -o option, you won’t see the amount of amount of RAM allocated as kernel buffer memory. The other columns of output show the amount of memory allocated as shared memory (System V IPC shared memory, to be precise), additional non- specific buffer memory, and the amount of cached data in memory. The shared memory column should be disregarded because it is no longer used. The final option that might prove useful is the -s secs option, which causes free to redisplay its report every secs seconds. The following exam- ple shows free’s output immediately before and during a kernel compilation: $ free -s5 -m -o total used free shared buffers cached Mem: 503 496 7 0 0 253 Swap: 1027 0 1027 total used free shared buffers cached Mem: 503 495 8 0 0 253 Swap: 1027 0 1027 total used free shared buffers cached Mem: 503 494 8 0 0 243 Swap: 1027 0 1027 total used free shared buffers cached Mem: 503 489 14 0 0 244 Swap: 1027 0 1027 This example used the -m option to display the output in megabytes, the -o option to turn off the buffer adjustment, and the -s5 option to refresh the display Performance Monitoring 807 40_599496 ch32.qxd 8/30/05 7:14 PM Page 807 every five seconds. The kernel compilation started between the first and second updates. One of the features you’ll notice in the bold-faced section is that the amount of cached data fell when the kernel build process started. Presumably, this occurred because data the kernel needed had to be read from disk, forcing a certain amount of cached data to be flushed. vmstat digs deeper into memory usage than free and pays particular attention to virtual memory (swap) usage. If your system is constantly swap- ping, disk I/O will slow to a crawl and the system will seem slow to respond to user input. vmstat makes it possible for you to detect the problem. You can then use top or one of the other utilities discussed in this chapter to identify what is causing the excessive swapping. First, however, vmstat’s syntax, bearing in mind that this discussion ignores options not related to virtual memory: vmstat [-S k|K|m|M] [-a] [-n] [secs [cnt]] vmstat [-S k|K|m|M] -m vmstat [-S k|K|m|M] -s To change the display unit, which defaults to bytes, use -S k for units of 1000 bytes, -S K for true kilobytes (1024 bytes), -S m for units of 1,000,000 bytes, or -S M for true megabytes (1,048,576 bytes). The examples in the text use -S K. Certain vmstat reports can be refreshed every secs seconds and, if cnt is specified, will refresh cnt times every secs seconds before vmstat terminates. In its simplest usage, vmstat’s output looks like the following: $ vmstat -S K procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 852 16132 0 249232 0 0 33 26 10 82 94 2 4 0 This information shows only the average usage since the system was booted. To get information about current usage, you must request a refreshing display using secs and, if you multiple reports, cnt, as shown in the follow- ing example: $ vmstat -S K 5 5 procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy id wa 3 0 852 4880 0 250120 0 0 33 26 15 135 93 3 4 0 2 0 852 14628 0 250308 0 0 0 30 1132 405 94 6 0 0 3 0 852 14168 0 250444 0 0 0 31 1131 418 93 7 0 0 4 0 852 6780 0 250528 0 0 0 35 1130 375 94 6 0 0 4 0 852 11856 0 247484 0 0 0 156 1149 422 93 7 0 0 808 Chapter 32 40_599496 ch32.qxd 8/30/05 7:14 PM Page 808 What information is shown? In the procs section, the r column shows the number of processes that are ready to run and waiting for their turn to run on the CPU and the b column shows the number of processes that are blocked, or sleeping, and thus not ready to run. In the first example, therefore, two processes are ready to run and waiting for CPU time and no processes are blocked. The four columns under the memory heading, show the following information: ■■ swpd — The amount of virtual memory in use ■■ free — The amount of physical RAM not in use ■■ buff — The amount of physical RAM used as buffers ■■ cache — The amount of physical RAM used as cache As you can see in the second example, created during a kernel compile, the amount of free and cache memory fluctuates constantly; the more active the system, the greater the fluctuation. If you specify -a, inact and active replace the buff and cache columns under the memory heading. inact displays the amount of inactive memory and active displays the amount of active memory. Inactive memory is the buffer memory the free command shows as free (unused) when buffer adjustments are enabled; active memory is memory that is both allocated and in use and maps to the used buffer memory reported by the free command. The following vmstat example shows the effect of the -a option: $ vmstat -S K -a 5 5 procs memory swap io system cpu r b swpd free inact active si so bi bo in cs us sy id wa 4 0 1500 13132 114240 346908 0 0 33 26 17 143 93 3 4 0 3 0 1500 14412 114308 345512 0 0 71 116 1172 486 91 9 0 0 4 0 1500 10704 114400 349152 0 0 22 36 1220 849 92 8 0 0 4 0 1500 20944 114416 338936 0 0 10 82 1145 521 94 6 0 0 3 0 1500 12240 114484 347584 0 0 13 49 1342 1437 92 8 0 0 Under the swap heading, si shows the amount of memory that has been read in from the swap device (or devices, if there are multiple swap files or partitions) and so the amount of memory that has been written out to a swap device. As you can see in the example just shown, swap usage on this system, even during a kernel compile, is negligible. In the io section, the columns bi and bo show the number of disk blocks (in units of 1024 bytes) read from and written to, respectively, the system’s block devices. Under the system heading, in lists the number of interrupts received per second and cs shows the number of context switches per second. Values under the cpu heading, finally, show the disposition of CPU usage, with each column expressed as a percentage of total CPU time (due to rounding, the val- ues might not add to 100 percent). The specific columns are: Performance Monitoring 809 40_599496 ch32.qxd 8/30/05 7:14 PM Page 809 ■■ us — The percentage of CPU time spent running user, or nonkernel, code ■■ sy — The percentage of time spent executing system, or kernel, code ■■ id — The percentage of CPU time that the CPU is idle ■■ wa — The percentage of CPU time spent waiting for I/O to complete Examining Kernel Memory Usage The memory usage information discussed so far examined memory from the point of view of the user or running processes. You haven’t seen with any amount of detail how the kernel itself is using memory. The last vmstat option, -m, gives you a window into the kernel’s internal memory usage. The -m option causes vmstat to display kernel slab usage. Slabs are caches of frequently used kernel memory objects, such as inodes, directory entries, file pointers, and ran- dom blocks of memory of specific sizes, such as 8192 bytes, 4096 bytes, and so on. Rather than use vmstat to view slab usage, however, you should use slabtop, which does for slabs what the top command does for processes, namely, show slab usage in a real-time updated format. Slabtop’s syntax is: slabtop [-d secs] [-s sort] [-o] -d secs specifies the number of seconds to pause between updates. -o tells slabtop to display its output once and then exit. The -s sort option sets the sort order, which defaults to the number of slab objects descending order, for the displayed slabs to sort. sort can be one of the values listed in Table 32-1. Table 32-1 slabtop Sorting Criteria CRITERIA ORDER DESCRIPTION a Ascending Sort by the number of active objects b Ascending Sort by the number of objects per slab c Descending Sort by cache size l Descending Sort by the number of slabs v Descending Sort by the number of active slabs n Ascending Sort by the slab name o Descending Sort by the number of objects (this the default sort order) p Descending Sort by the number of pages per slab s Descending Sort by the object size u Descending Sort by cache utilization 810 Chapter 32 40_599496 ch32.qxd 8/30/05 7:14 PM Page 810 Figure 32-1 Viewing slabtop’s default output. Invoked with no options, slabtop’s output resembles Figure 32-1. The slab cache listing, updated every three seconds by default, shows detailed slab cache information. The top five lines show summary information for the number of individual objects, the total number of slabs containing objects, the number of slab caches, and slab size statistics. The bottom portion of the display shows the specifics for each type of slab cache sorted in descend- ing order by the object type. You can change the sort order at runtime by press- ing the key associated with the sort criteria that interests you (see Table 32-1). TIP The p sort option for sorting slabtop’s output by the number of pages per slab does not appear to function in slabtop version 3.2.3. However, you can view this information using the following sort invocation: $ sort -k6,6 -nr < /proc/slabinfo | cut -f1 -d: size-131072(DMA) 0 0 131072 1 32 size-131072 0 0 131072 1 32 size-65536(DMA) 0 0 65536 1 16 size-65536 4 4 65536 1 16 size-32768(DMA) 0 0 32768 1 8 size-32768 49 49 32768 1 8 size-16384(DMA) 0 0 16384 1 4 size-16384 3 3 16384 1 4 tcpv6_sock 1 5 1376 5 2 task_struct 120 120 1392 5 2 The sixth column shows the number of pages per slab. To view the output sorted in ascending order, omit -r: $ sort -k6,6 -n < /proc/slabinfo | cut -f 1 -d: # name <active_objs> <num_objs> <objsize> <objperslab> Performance Monitoring 811 40_599496 ch32.qxd 8/30/05 7:14 PM Page 811 <pagesperslab> slabinfo - version anon_vma 3069 3213 32 119 1 arp_cache 2 20 192 20 1 as_arq 0 0 64 61 1 avc_node 12 600 52 75 1 bdev_cache 14 18 608 6 1 bio 287 287 96 41 1 biovec-1 293 452 16 226 1 biovec-16 260 260 192 20 1 Again, the sixth column shows the number of pages per slab. If you run slabtop on a kernel that was compiled with the configuration option CONFIG_DEBUG_SLAB enabled, you will see additional slab cache statis- tics. The first line of the output will include (statistics) and the real-time display will show five additional columns: ■■ The maximum number of active objects in the slab ■■ The number of times objects have been allocated ■■ The number of times new pages have added to the cache (cache growth) ■■ The number of times unused pages have been removed from the cache (cache reaping) ■■ The number of errors allocating new pages to the cache Unless you run a debugging kernel and are actively working on the kernel, you won’t need this additional information. Nevertheless, you will at least know how to produce this information if someone asks for it. Viewing Running Tasks In many cases, you will be less concerned about how much memory a process is using and more concerned about what processes are running, or perhaps more likely, what processes are running out of control. The canonical tools for viewing running processes are ps and top. ps gives you a snapshot view of the currently active processes, and top gives you a real-time updated display of running processes. 812 Chapter 32 40_599496 ch32.qxd 8/30/05 7:14 PM Page 812 Getting Started with ps The implementation of ps that is used on Linux systems (from the procps suite) is a classic example of a Linux command gone horribly wrong. It has far too many options, a number of which are redundant. On the other hand, it lacks built-in email functionality, so ps might yet be salvageable. Seriously, ps is a powerful tool for viewing the current process list. Refer to the section titled “Obtaining Process Information” in Chapter 28 for discussions about addi- tional process management programs. Tables 32-2 through 32-5 borrow the layout of ps’s syntax description from its manual page and organize each group of options into tables based on the options’ purpose. ps supports both Unix98 options, which are preceded by a hyphen (-), and BSD options, which lack the initial Where the functionality is identical or very similar, the BSD options have been omitted. In some cases, apparently identical Unix98 and BSD are listed because the BSD option shows different output from the similarly invoked Unix98 option. The options from this list that you’ll most likely use are -e to select all process, r to select only running processes, and -N to reverse the meaning of the selection criteria. -N helps you express selection criteria for which there are no command line options. For example, if you want to see all processes except those owned by the users bubba and root (see the -u, -U, and U options in Table 32-3), you might use the following command: $ ps U bubba U root -N PID TTY STAT TIME COMMAND 3947 ? Ss 0:00 portmap 4142 ? SLs 0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid 4179 ? Ss 0:03 xfs -droppriv -daemon 4217 ? Ss 0:00 dbus-daemon-1 system Table 32-2 Basic Process Selection OPTION DESCRIPTION -N Negates the selection criteria specified with other options -a Selects all processes with a TTY except session leaders -d Selects all except session leaders -e Selects all processes T Selects all processes on the invoking terminal r Selects only running processes x Selects processes without controlling TTYs Performance Monitoring 813 40_599496 ch32.qxd 8/30/05 7:14 PM Page 813 [...]... blocks read is 4,0 0 1,6 4 3, which totals 2,0 4 8,8 4 1,2 16 bytes or about 1.9 GB ■ ■ The total number of blocks written is 3,4 2 3,9 8 1, which totals 1,7 5 3,0 7 8,2 72 bytes or about 1.6 GB 823 824 Chapter 32 The block size iostat uses is shown in terms of disk sectors Thus, when you are analyzing a physical disk, blocks are 512 bytes, but when you are looking at a disk partition, the measurement appears in kilobytes... For example, the more read and write requests that are merged (see rrqm/s, wrqm/s, r/s, and w/s ), the more efficient the I/O subsystem is because read and write requests are being batched together and filled all at once On the other hand, if too many requests are being merged, I/O throughput, the amount of data actually being read from or written to disk, can decrease Throughput is measured by sectors... processes that are ready to run but that cannot because they are waiting for CPU time You can see in the example output that the run queue is fairly steady at an average of 10 processes waiting to run of a total of 143 processes However, the load averages are gradually increasing, indicating that overall system usage is high and constant, rather than spiking temporarily The -u option, finally, shows you what... singlepurpose tools, a certain utility and convenience exists in being able to access all of a system s performance metrics using one program This section discusses sar (pronounced like “car” ), which stands for system activity report sar is a popular tool that provides a single interface for collecting, storing, and analyzing system monitoring and performance data sar and sadc are, like iostat, installed... sa2 command, which is a shell command that invokes sar using the -f option See /usr/lib/sa/sa2 for more details Because sar can be used to monitor multiple subsystems, the text looks at each subsystem separately Before diving in to the subsystem-specific modes, however, you’ll want to know some of the general command line options that sar supports Probably the most useful option is -f ifile, which lets... Memory and CPU usage serve as important indicators of your system s health and its overall performance and efficiency, but they are not the only measures A common truism in performance-tuning literature and practice is that your system is only as fast as its slowest component In most systems, the slowest component is the I/O subsystem Memory, CPU, bus, and network speeds long ago surpassed the capability... Unfortunately, sysstat is not installed as part of some of the default Fedora Core installation profiles, notably the Fedora Core workstation profile If you want to use sar, make sure the sysstat package is installed On Fedora Core and RHEL system, sar works in conjunction with the sadc program that is started at boot time by the sysstat service sadc, which stands for system activity data collector, samples... speeds Accordingly, you need a good tool that helps you identify and isolate where I/O bottlenecks are occurring iostat, discussed in this section, and sar, discussed in the next section, are just the tools to use How you proceed after you’ve fingered the performance culprit is a different issue, of course; diagnostics identify only the problem N OT E In order to use the iostat command, the sysstat package... output format produced by j, consisting of only the PID, the process group ID (PGID ), the session ID (SID ), the TTY on which the program is running, the cumulative CPU time, and the bare command name In many situations, the Unix98 output will be all you need The BSD output is much more informative (at least in this case ), showing all of the processes owned by the current user, not just the processes... summarize, sar is the Swiss army knife of system monitoring utilities Unlike the other programs and command discussed in this chapter, sar usually displays reports using data collected and stored in the file system As a result, sar’s impact on system performance is minimal If you run it as shown in this section, specifying an interval between updates and/or the number of updates to display, expect . number of blocks read is 4,0 0 1,6 4 3, which totals 2,0 4 8,8 4 1,2 16 bytes or about 1.9 GB. ■■ The total number of blocks written is 3,4 2 3,9 8 1, which totals 1,7 5 3,0 7 8,2 72 bytes or about 1.6 GB. Performance. display unit, which defaults to bytes, use -S k for units of 1000 bytes, -S K for true kilobytes (1024 bytes ), -S m for units of 1,0 0 0,0 00 bytes, or -S M for true megabytes ( 1,0 4 8,5 76 bytes) inodes, directory entries, file pointers, and ran- dom blocks of memory of specific sizes, such as 8192 bytes, 4096 bytes, and so on. Rather than use vmstat to view slab usage, however, you should