More Bash Shell Commands

One of the toughest jobs of being a Linux system administrator is keeping track of what’s running on the system — especially now, when graphical desktops take a handful of programs just to produce a single desktop. There are always a lot of programs running on the system.

Fortunately, there are a few command line tools that can help make life easier for you. This section covers a few of the basic tools you’ll need to know how to use to manage programs on your Linux system.

Peeking at the processes

When a program runs on the system, it’s referred to as a process. To examine these processes, you’ll need to become familiar with the ps command, the Swiss Army knife of utilities. It can produce lots of information about all the programs running on your system.

Unfortunately, with this robustness comes complexity — in the form of numerous parameters — making the ps command probably one of the most difficult commands to master. Most system administrators find a subset of these parameters that provide the information they want, and then stick with using only those.

That said, however, the basic ps command doesn’t really provide all that much information:

$ ps
  PID TTY TIME CMD
  3081 pts/0 00:00:00 bash
  3209 pts/0 00:00:00 ps
  $

Not too exciting. By default the ps command shows only the processes that belong to the current user and that are running on the current terminal. In this case, I only had my bash shell running (remember, the shell is just another program running on the system) and, of course, the ps command itself.

The basic output shows the process ID (PID) of the programs, the terminal (TTY) that they are running from, and the CPU time the process has used.

The GNU ps command that’s used in Linux systems supports three different types of command line parameters:

Unix-style parameters, which are preceded by a dash

• BSD-style parameters, which are not preceded by a dash
• GNU long parameters, which are preceded by a double dash

The following sections examine the three different parameter types and show examples of how they work.

Unix-style parameters

The Unix-style parameters originated with the original ps command that ran on the AT&T Unix systems invented by Bell Labs. These parameters are shown in Table below.

That’s a lot of parameters, and remember, there are still more! The key to using the ps command is not to memorize all of the available parameters, only those you find most useful. Most Linux system administrators have their own sets of commonly used parameters that they remember for extracting pertinent information. For example, if you need to see everything running on the system, use the -ef parameter combination (the ps command lets you combine parameters together like this):

$ ps -ef
  UID PID PPID C STIME TTY TIME CMD
  root 1 0 0 11:29 ? 00:00:01 init [5]
  root 2 0 0 11:29 ? 00:00:00  [kthreadd]
  root 3 2 0 11:29 ? 00:00:00  [migration/0]
  root 4 2 0 11:29 ? 00:00:00  [ksoftirqd/0]
  root 5 2 0 11:29 ? 00:00:00  [watchdog/0]
  root 6 2 0 11:29 ? 00:00:00  [events/0]
  root 7 2 0 11:29 ? 00:00:00  [khelper]
  root 47 2 0 11:29 ? 00:00:00  [kblockd/0]
  root 48 2 0 11:29 ? 00:00:00  [kacpid]
  68 2349 1 0 11:30 ? 00:00:00 hald
  root 2489 1 0 11:30 tty1 00:00:00  /sbin/mingetty tty1
  root 2490 1 0 11:30 tty2 00:00:00  /sbin/mingetty tty2
  root 2491 1 0 11:30 tty3 00:00:00  /sbin/mingetty tty3
  root 2492 1 0 11:30 tty4 00:00:00  /sbin/mingetty tty4
  root 2493 1 0 11:30 tty5 00:00:00  /sbin/mingetty tty5
  root 2494 1 0 11:30 tty6 00:00:00  /sbin/mingetty tty6
  root 2956 1 0 11:42 ? 00:00:00 /usr/sbin/httpd
  apache 2958 2956 0 11:42 ? 00:00:00  /usr/sbin/httpd
  apache 2959 2956 0 11:42 ? 00:00:00  /usr/sbin/httpd
  root 2995 1 0 11:43 ? 00:00:00  auditd
  root 2997 2995 0 11:43 ? 00:00:00  /sbin/audispd
  root 3078 1981 0 12:00 ? 00:00:00  sshd: rich [priv]
  rich 3080 3078 0 12:00 ? 00:00:00  sshd: rich@pts/0
  rich 3081 3080 0 12:00 pts/0  00:00:00 -bash
  rich 4445 3081 3 13:48 pts/0  00:00:00 ps -ef
  $

I’ve cut out quite a few lines from the output to save space, but as you can see, there are lots of processes running on a Linux system. This example uses two parameters, the -e parameter, which shows all of the processes running on the system, and the -f parameter, which expands the output to show a few useful columns of information:

 UID: The user responsible for launching the process

• PID: The process ID of the process
• PPID: The PID of the parent process (if a process is started by another process)
• C: Processor utilization over the lifetime of the process
• STIME: The system time when the process started
• TTY: The terminal device from which the process was launched
• TIME: The cumulative CPU time required to run the process
• CMD: The name of the program that was started 

This produces a reasonable amount of information, which is what many system administrators would like to see. For even more information, you can use the -l parameter, which produces the long format output:

$ ps -l
  F S UID PID PPID C PRI NI ADDR SZ  WCHAN TTY TIME CMD
  0 S 500 3081 3080 0 80 0 - 1173 wait  pts/0 00:00:00 bash
  0 R 500 4463 3081 1 80 0 - 1116 -  pts/0 00:00:00 ps
  $

Notice the extra columns that appear when you use the -l parameter:

F: System flags assigned to the process by the kernel

Before moving on, there’s one more extremely handy parameter to remember, -H. The -H parameter organizes the processes in a hierarchical format, showing which processes started which other processes. Here’s an extraction from an -efH-formatted listing:

$ ps -efH
  UID PID PPID C STIME TTY TIME CMD
  root 3078 1981 0 12:00 ? 00:00:00 sshd:  rich [priv]
  rich 3080 3078 0 12:00 ? 00:00:00  sshd: rich@pts/0
  rich 3081 3080 0 12:00 pts/0  00:00:00 -bash
  rich 4803 3081 1 14:31 pts/0  00:00:00 ps –efH

Notice the shifting in the CMD column output. This shows the hierarchy of the processes that are running. First, the sshd process started by the root user (this is the Secure Shell (SSH) server session, which listens for remote SSH connections). Next, when I connected from a remote terminal to the system, the main SSH process spawned a terminal process (pts/0), which in turn spawned a bash shell.

From there, I executed the ps command, which appears as a child process from the bash process.
On a multi-user system, this is a very useful tool when trying to troubleshoot runaway processes, or when trying to track down which userid or terminal they belong to.

BSD-style parameters

Now that you’ve seen the Unix parameters, let’s take a look at the BSD-style parameters. The  Berkeley Software Distribution (BSD) was a version of Unix developed at (of course) the University of California, Berkeley. It had many subtle differences from the ATT Unix system, thus sparking many Unix wars over the years. The BSD version of the ps command parameters are shown in Table below.

As you can see, there’s a lot of overlap between the Unix and BSD types of parameters. Most of the information you can get from one you can also get from the other. Most of the time which one you use depends on which format you’re more comfortable with (for example, if you were used to a BSD environment before using Linux).

When you use the BSD-style parameters, the ps command automatically changes the output to simulate the BSD format. Here’s an example using the l parameter:

$ ps l
  F UID PID PPID  PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
  0 500 3081 3080  20 0 4692 1432 wait Ss pts/0 0:00 -bash
  0 500 5104 3081  20 0 4468 844 - R+ pts/0 0:00 ps l
  $

Notice that while many of the output columns are the same as when we used the Unix-style parameters, there are a couple of different ones:

• VSZ: The size in kilobytes of the process in memory
• RSS: The physical memory that a process has used that isn’t swapped out
• STAT: A two-character state code representing the current process state

Many system administrators like the BSD-style l parameter because it produces a more detailed state code for processes (the STAT column). The two-character code more precisely defines exactly what’s happening with the process than the single-character Unix-style output.  The first character uses the same values as the Unix-style S output column, showing when a process is sleeping, running, or waiting. The second character further defines the process’s status:

• : The process is running at high priority.
• N: The process is running at low priority.
• L: The process has pages locked in memory.
• s: The process is a session leader.
• l: The process is multi-threaded.
• +: The process is running in the foreground.

From the simple example shown above, you can see that the bash command is sleeping, but it is a session leader (it’s the main process in my session), while the ps command was running in the foreground on the system.

The GNU long parameters

Finally, the GNU developers put their own touches on the new, improved ps command by adding a few more options to the parameter mix. Some of the GNU long parameters copy existing Unix- or BSD-style parameters, while others provide new features. Table below lists the GNU long parameters available.

You can combine GNU long parameters with either Unix- or BSD-style parameters to really customize your display. One cool feature of GNU long parameters that I really like is the –forest parameter. It displays the hierarchical process information, but using ASCII characters to draw cute charts:

1981 ? 00:00:00  sshd
  3078 ? 00:00:00   sshd
  3080 ? 00:00:00   sshd
  3081 pts/0  00:00:00  bash
  16676 pts/0  00:00:00  ps

This format makes tracing child and parent processes a snap!

Real-time process monitoring

The ps command is great for gleaning information about processes running on the system, but it has one drawback. The ps command can only display information for a specific point in time. If you’re trying to find trends about processes that are frequently swapped in and out of memory, it’s hard to do that with the ps command.

Instead, the top command can solve this problem. The top command displays process information similarly to the ps command, but it does it in real-time mode. Figure below is a snapshot of the top command in action.
 
 The output of the top command while it is running

The first section of the output shows general system information. The first line shows the current time, how long the system has been up, the number of users logged in, and the load average on the system. The load average appears as three numbers, the 1-minute, 5-minute, and 15-minute load averages. The higher the values, the more load the system is experiencing. It’s not uncommon for the 1-minute load value to be high for short bursts of activity. If the 15-minute load value is high, your system may be in trouble.

The second line shows general process information (called tasks in top): how many processes are running, sleeping, stopped, and zombie (have finished but their parent process hasn’t responded).  The next line shows general CPU information. The top display breaks down the CPU utilization into several categories depending on the owner of the process (user versus system processes) and the state of the processes (running, idle, or waiting).

Following that, there are two lines that detail the status of the system memory. The first line shows the status of the physical memory in the system, how much total memory there is, how much is currently being used, and how much is free. The second memory line shows the status  of the swap memory area in the system (if any is installed), with the same information.

Finally, the next section shows a detailed list of the currently running processes, with some information columns that should look familiar from the ps command output:

• PID: The process ID of the process
• USER: The user name of the owner of the process
• PR: The priority of the process
• NI: The nice value of the process
• VIRT: The total amount of virtual memory used by the process
• RES: The amount of physical memory the process is using
• SHR: The amount of memory the process is sharing with other processes
• S: The process status (D = interruptible sleep, R = running, S = sleeping, T = traced or stopped, or Z = zombie)
• %CPU: The share of CPU time that the process is using
• %MEM: The share of available physical memory the process is using
• TIME+: The total CPU time the process has used since starting
• COMMAND: The command line name of the process (program started) 

By default, when you start top it sorts the processes based on the %CPU value. You can change the sort order by using one of several interactive commands while top is running. Each interactive command is a single character you can press while top is running and changes the behavior of the program. These commands are shown in Table below.

You have lots of control over the output of the top command. Using this tool, you can often find offending processes that have taken over your system. Of course, once you find one, the next job is to stop it, which brings us to the next topic.

Stopping processes

A crucial part of being a system administrator is knowing when and how to stop a process. Sometimes a process gets hung up and just needs a gentle nudge to either get going again or stop. Other times, a process runs away with the CPU and refuses to give it up. In both cases, you need a command that will allow you to control a process. Linux follows the Unix method of interprocess communication.

In Linux, processes communicate between each other using signals. A process signal is a predefined message that processes recognize and may choose to ignore or act on. The developers program how a process handles signals. Most well-written applications have the ability to receive and act on the standard Unix process signals. These signals are shown in Table below. There are two commands available in Linux that allow us to send process signals to running processes.
Table5

The kill command

The kill command allows you to send signals to processes based on their process ID (PID). By default the kill command sends a TERM signal to all the PIDs listed on the command line.  Unfortunately, you can only use the process PID instead of its command name, making the kill command difficult to use sometimes.

To send a process signal, you must either be the owner of the process or be logged in as the root user.

$ kill 3940
  -bash: kill: (3940) - Operation not permitted
  $

The TERM signal tells the process to kindly stop running. Unfortunately, if you have a runaway process, most likely it’ll ignore the request. When you need to get forceful, the -s parameter allows you to specify other signals (either using their name or signal number).The generally accepted procedure is to first try the TERM signal. If the process ignores that, try the INT or HUP signals. If the program recognizes these signals, it’ll try to gracefully stop doing what it was doing before shutting down. The most forceful signal is the KILL signal. When a process receives this signal, it immediately stops running. This can lead to corrupt files.

As you can see from the following example, there’s no output associated with the kill command.
# kill -s HUP 3940
#

To see if the command was effective, you’ll have to perform another ps or top command to see if the offending process stopped.

The killall command

The killall command is a powerful way to stop processes by using their names rather than the PID numbers. The killall command allows you to use wildcard characters as well, making it a very useful tool when you’ve got a system that’s gone awry.