About this series
The typical UNIX® administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment.
 |
|
There are many different potential causes of a slow system, but they can generally be slotted into the following areas:
- Too many processes. Your system is merely running too many applications at the same time, or running a small number of very CPU-intensive operations. Either you are overloading the server, or there is a runaway process that is sapping your system resources.
- Too much active memory. If your processes are using a lot of memory, then your system might be swapping a lot of pages in and out to disk and, that means, your system is spending more time swapping the memory than it is actually using it.
- Hardware fault. Occasionally you will come across a hardware fault that causes slow downs. A bad network card, hard drive, or memory might result in your system spending a long time waiting for information.
To diagnose the problem, you need to use a number of available tools to examine your UNIX system.
|
|
If your machine is running particularly slow, then the first issue will be how you connect to your machine so that you can start the monitoring process. A slow machine might not accept connections through Telnet or over a remove shell protocol, such as ssh.
If you are not already logged in, getting access might be impossible. Instead, consider using the console, either directly or through a separate hardware solution, such as a network or serial based console monitor.
The console is more likely to allow you to log in, because there will already be a login process (which will be replaced with your shell) running. If, once you log in, you are unable to run any processes through your shell, it indicates that your system has run out of process space; a reboot will probably be the only way of returning your system to normal.
To reboot your system, use init or telinit to adjust the run level; run level 6 is usually synonymous with a reboot. Using init/telinit is more likely to reboot the system, as there is only one process involved to force the reboot.
Once the system is back up and running, you need to use some of the techniques in this article to monitor the running state of the system and record the output. If the slow system event happens again, you can then perform a post-mortem debug and examine the reason why your system runs so slowly.
|
|
If you suspect that your machine is running slowly, then the first command you should run is uptime. Uptime reports the current time, the amount of time the machine has been up and running (in other words, the time since the machine has booted), and the current number of users. It then provides three figures that show the load average in the last minute, five minutes, and fifteen minutes. For example:
|
In this example, the machine has had a load average of more than two in the last minute, more than five in the last five minutes, and more than three in the last fifteen minutes.
The definition of load average is complex and driven by the status of the processes that are executing. In general, each process that is running, waiting for CPU, or waiting on I/O would add one to the load average. These figures are calculated and then averaged over time.
In a single-CPU system, a load average of over one means that the CPU is under powered for the type of load you are supplying. But because of the multi-process nature of UNIX, it is generally acceptable for a load average of up to two over the long term (in other words, the fifteen minute figure) before you should be concerned.
In a multi-CPU (or multi-core) system, divide the load average by the number of CPUs. To determine if the machine is over stressed, use the same principles as above.
An alternative way of looking at the figures is to treat them as a percentage; in other words, if the figures above were from a single-CPU system, then the machine would be able to cope with the load if it was 224 percent faster.
Within a multi-CPU system, you should use the number of CPUs plus one to determine the maximum load. For example, a four-CPU system would have a maximum load average of 5.
It is quite common to find a machine that has a load average that is significantly higher than its maximum for a short period. For example, when building or compiling an application or performing a very disk intensive task, your load averages can shoot up. This is why the output includes one, five, and fifteen minute averages, because it helps to smooth out any transient extreme loads.
Any long-term, or unexpected, high values probably signify a problem and need further investigation. If the numbers are low but your system is slow, it might point to a swap space issue.
|
|
If you are managing a large network of systems, then there is a simple way of monitoring the load and usage of all the machines in your network. The ruptime tool collects data that is broadcast by all the machines on the network and collects it into a local file so that the current status of all the machines can be examined.
For example, Listing 1 shows the output from a small network:
Listing 1. Output from a small network
|
The last machine has not reported any data for eleven minutes, so it is officially listed as down.
To generate the information, the rwhod daemon (sometimes in.rwhod) needs to be running on each machine in your local network. This daemon broadcasts the information for the local machine and collects the broadcast data from all the other machines.
Because of the way the rwho/ruptime system works, there can be performance issues, especially in very large networks where the number of systems reporting and the network traffic they generate can be considered to be detrimental. In very busy systems, the need to broadcast the data can also mean that the information is never reported, the data can be out of date, or the system could be reported as down when it is just busy.
|
|
If you suspect that the problem is due to a large or overly busy process, then you should check the output of the ps tool, look for the process size, the percentage of memory, and CPU being utilized. On an SVR4 system (Solaris and AIX®), you can use the following command to get a list of processes (see Listing 2).
Listing 2. Command to get a list of processes
|
Listing 3 shows how it looks on a BSD-derived system.
Listing 3. Getting a list of processes on a BSD system
|
In both cases, the CPU and memory usage percentages have been displayed in the process list so that you can get a good idea of the loading on the system. The 's' and 'stat' columns (for SVR4 and BSD, respectively) show the current status of the process. A large number of running processes (status 'R') indicate that the process is currently running.
By using a combination of the state, CPU, and memory percentage, you should be able to identify if there is a process that is running away and over using your system resources.
|
|
The iostat tool provides information about the terminal, disk activity, and CPU utilization. You can specify a single numerical argument to set the report interval; a second numerical argument sets the number of reports. For example, Listing 4 shows how to report the statistics every five seconds.
Listing 4. Reporting statistics every five seconds
|
The exact information, shown by default, varies from system to system; Listing 4 was from a Solaris system. The example in Listing 5 comes from a BSD environment.
Listing 5. iostat in a BSD environment
|
Dealing with the CPU statistics first, the columns show user (us), system (sy), and idle (id) percentages. The user time shows how much time was spent on user processes. The system time shows the time in system processes (including, when wait time is not shown, the time the system is waiting for I/O). The idle time shows the percentage of time that the CPU(s) were idle.
Disk output shows how busy the individual physical disks (including NFS mounts, if applicable) are, usually in transactions per second and MB or KB transferred per second. Large figures here, especially if combined with high wait/system times, might indicate a disk that is too slow for the system. You can try to spread your application so that it uses different disks you might be able to improve performance.
If the disk is the same as the one used for virtual memory, then it might be a problem with lack of memory and too much swapping.
|
|
You can monitor virtual memory statistics with the vmstat tool. Like iostat, it accepts a numerical interval (see Listing 6).
Listing 6. Monitoring memory statistics with vmstat
|
The vmstat tool outputs thread/process information, memory/swap usage, page ins/outs, disk I/O, page faults, and CPU statistics.
The CPU/thread block shows the processes/threads in the run queue (r), blocked processes waiting for I/O resources (b), and those that were swapped. High numbers in the blocked processes column indicates slow disks. High numbers in the swapped column indicate that there are two many processes using too much memory that need to be swapped in and out. Swapping is an expensive process and will significantly degrade your system.
The memory shows the amount of swap currently available and the size of the free list (the number of pages that could be swapped if the RAM were requested). Low swap values indicate that you are running out of swap space, which doesn't necessarily show a problem, as long as you have enough RAM to run the applications. Low free list values might indicate that you have a lot of active RAM in use, which might trigger swap space use if you add more processes to the system.
The page columns show the pages of memory swapped in and out to disk. The key columns are the pi/po (page in/page out), which indicate how many pages have been exchanged. High paging indicates a lack of RAM; a high scan rate (the sr column) shows a potential memory bottleneck.
|
|
The top tool can provide a useful way to monitor a live system and the active processes, loading, and memory statistics. There are many different types of top, some of which are installed by default on some systems and also the latest open source version of the tool. The information provided is like a combination of the uptime, swap space, and ps tools. For example, the following is from a Solaris system running V3.5.1 of the top tool (see Listing 7).
Listing 7. Using top
|
The top tool shows the CPU usage of individual processes; for example, in the previous sample, you can see that you are compiling a number of files and how much CPU they are using.
You should also keep an eye on the process state: high numbers of running processes might indicate an over-busy system (compare the running processes with the CPU states and load averages for the system).
Top itself can be a CPU hog; it is best to run it at a relatively large update interval, to prevent the monitoring having a detrimental effect on the performance of your system. You can specify the update interval in seconds using either -s or -d command-line option (depending on your platform).
|
|
In situations where you cannot monitor the status of your server live but you want to be able to monitor the status of a machine after it has caused a problem, you can use the SAR (system activity reporter) tool. This works by recording information at specific intervals into a global file that can then be post processed to display information about the machine.
Because the recording process continues in the background, it can be used to plot the system performance over time and might help you identify the cause of the problem. Information is generally recorded day by day, for up to a month, at intervals that you specify. Logs are written into /var/log/sa/saDD or /usr/adm/sa/saDD, where DD is the day of the month. Enabling SAR is specific to the system, and generally you will need to set up a cron job that automatically runs the collection script (sa1). Another script, sa2, can create a daily report that you can study. For example, the crontab below shows the default system performance statistics recording on a Solaris system:
|
Once the information has been collected, you can extract data using the sar command. The amount of information, which is recorded by the system, can be voluminous, and the level of detail that can be selected and extracted from that data is similarly large. However, you can get a feel for the quantity and quality of the data by using the -A command-line argument to SAR, which reports all the currently recorded information.
Listing 8. Output generated with the sar command, using the -A argument
|
Where possible, the output above has been trimmed to limit the amount of data shown (not all disk stats are shown, for example). For more information on SAR, check the Resources section and the manual page for your system.
|
|
While there is not always a direct correlation between a slow UNIX system and the statistical information that you can extract, your first job when discovering a slow system should be to collect as much information as possible. Whether you do this actively (through ps, uptime and other tools) or passively (through SAR or top) depends on the situation. Armed with that information, you should be able to tell whether your UNIX system is slow because it is overworked (CPU over usage), low on physical memory (large amounts of swapping), or there is a problem with a runaway process (high single process CPU time).