Starting with Linux Shells
Before you can dive into working with the Linux command line and shells, it’s a good idea to first understand what Linux is, where it came from, and how it works. This section walks you through what Linux is, and explains where the shell and command line fit in the overall Linux picture. What Is Linux?
files and programs. While Linus created If you’ve never worked with Linux before, you may be confused as to why there are so many different versions of it available. I’m sure that you have heard various terms such as distribution, LiveCD, and GNU when looking at Linux packages and been confused. Trying to wade through the world of Linux for the first time can be a tricky experience. This section will take some of the mystery out of the Linux system before we start working on commands and scripts.
For starters, there are four main parts that make up a Linux system:
- The Linux kernel
- The GNU utilities
- A graphical desktop environment
- Application software
Each of these four parts has a specific job in the Linux system. Each of the parts by itself isn’t very useful. Figure below shows a basic diagram of how the parts fit together to create the overall Linux system.
Figure: The Linux system
This section describes these four main parts in detail, and gives you an overview of how they work together to create a complete Linux system.
Looking into the Linux kernel
The core of the Linux system is the kernel. The kernel controls all of the hardware and software on the computer system, allocating hardware when necessary, and executing software when required.
If you’ve been following the Linux world at all, no doubt you’ve heard the name Linus Torvalds.
Linus is the person responsible for creating the first Linux kernel software while he was a student at the University of Helsinki. He intended it to be a copy of the Unix system, at the time a popular operating system used at many universities.
After developing the Linux kernel, Linus released it to the Internet community and solicited suggestions for improving it. This simple process started a revolution in the world of computer operating systems. Soon Linus was receiving suggestions from students as well as professional programmers from around the world.
Allowing anyone to change programming code in the kernel would result in complete chaos. To simplify things, Linus acted as a central point for all improvement suggestions. It was ultimately Linus’s decision whether or not to incorporate suggested code in the kernel. This same concept is still in place with the Linux kernel code, except that instead of just Linus controlling the kernel code, a team of developers has taken on the task.
The kernel is primarily responsible for four main functions:
- System memory management
- Software program management
- Hardware management
- Filesystem management
The following sections explore each of these functions in more detail.
System memory management
One of the primary functions of the operating system kernel is memory management. Not only does the kernel manage the physical memory available on the server, it can also create and manage virtual memory, or memory that does not actually exist.
It does this by using space on the hard disk, called the swap space. The kernel swaps the contents of virtual memory locations back and forth from the swap space to the actual physical memory.
This allows the system to think there is more memory available than what physically exists (shown in Figure below).
The Linux system memory map
The memory locations are grouped into blocks called pages. The kernel locates each page of memory either in the physical memory or the swap space. The kernel then maintains a table of the memory pages that indicates which pages are in physical memory, and which pages are swapped out to disk.
The kernel keeps track of which memory pages are in use and automatically copies memory pages that have not been accessed for a period of time to the swap space area (called swapping out, even if there’s other memory available. When a program wants to access a memory page that has been swapped out, the kernel must make room for it in physical memory by swapping out a different memory page, and swap in the required page from the swap space. Obviously, this process takes time, and can slow down a running process. The process of swapping out memory pages for running applications continues for as long as the Linux system is running.
You can see the current status of the virtual memory on your Linux system by viewing the special /proc/meminfo file. Here’s an example of a sample /proc/meminfo entry:
# cat /proc/meminfo
MemTotal: 255392 kB
MemFree: 4336 kB
Buffers: 1236 kB
Cached: 48212 kB
SwapCached: 1028 kB
Active: 182932 kB
Inactive: 44388 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 255392 kB
LowFree: 4336 kB
SwapTotal: 524280 kB
SwapFree: 514528 kB
Dirty: 456 kB
Writeback: 0 kB
AnonPages: 176940 kB
Mapped: 40168 kB
Slab: 16080 kB
SReclaimable: 4048 kB
SUnreclaim: 12032 kB
PageTables: 4048 kB
NFS Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 651976 kB
Committed AS: 442296 kB
VmallocTotal: 770040 kB
VmallocUsed: 3112 kB
VmallocChunk: 766764 kB
HugePages Total: 0
HugePages Free: 0
HugePages Rsvd: 0
Hugepagesize: 4096 kB
the Mem: line shows that this Linux server has 256MB of physical memory. It also shows that about 4 MB is not currently being used (free). The output also shows that there is about 512MB of swap space memory available on this system.
By default, each process running on the Linux system has its own private memory pages. One process cannot access memory pages being used by another process. The kernel maintains its own memory areas. For security purposes, no processes can access memory used by the kernel processes.
To facilitate data sharing, you can create shared memory pages. Multiple processes can read and write to and from a common shared memory area. The kernel maintains and administers the shared memory areas and allows individual processes access to the shared area.
The special ipcs command allows you to view the current shared memory pages on the system. Here’s the output from a sample ipcs command:
# ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 0 rich 600 52228 6 dest
0x395ec51c 1 oracle 640 5787648 6
Each shared memory segment has an owner that created the segment. Each segment also has a standard Linux permissions setting that sets the availability of the segment for other users. The key value is used to allow other users to gain access to the shared memory segment.
Software program management
The Linux operating system calls a running program a process. A process can run in the foreground, displaying output on a display, or it can run in background, behind the scenes. The kernel controls how the Linux system manages all the processes running on the system.
The kernel creates the first process, called the init process, to start all other processes on the system. When the kernel starts, it loads the init process into virtual memory. As the kernel starts each additional process, it gives it a unique area in virtual memory to store the data and code that the process uses.
Some Linux implementations contain a table of processes to start automatically on bootup. On Linux systems this table is usually located in the special file /etc/inittabs. The Linux operating system uses an init system that utilizes run levels. A run level can be used to direct the init process to run only certain types of processes, as defined in the /etc/inittabs file. There are five init run levels in the Linux operating system.
At run level 1, only the basic system processes are started, along with one console terminal process. This is called single user mode. Single user mode is most often used for emergency filesystem maintenance when something is broken. Obviously, in this mode only one person (usually the administrator) can log in to the system to manipulate data.
The standard init run level is 3. At this run level most application software such as network support software is started. Another popular run level in Linux is run level 5. This is the run level where the system starts the graphical X Window software, and allows you to log in using a graphical desktop window.
The Linux system can control the overall system functionality by controlling the init run level. By changing the run level from 3 to 5, the system can change from a console-based system to an advanced, graphical X Window system. Later on you’ll see how to use the ps command to view the processes currently running on the Linux system. Here’s an example of what you’ll see using the ps command:
$ ps ax
PID TTY STAT TIME COMMAND
1 ? S 0:03 init
2 ? SW 0:00 [kflushd]
3 ? SW 0:00 [kupdate]
4 ? SW 0:00 [kpiod]
5 ? SW 0:00 [kswapd]
243 ? SW 0:00 [portmap]
295 ? S 0:00 syslogd
305 ? S 0:00 klogd
320 ? S 0:00 /usr/sbin/atd
335 ? S 0:00 crond
350 ? S 0:00 inetd
365 ? SW 0:00 [lpd]
403 ttyS0 S 0:00 gpm -t ms
418 ? S 0:00 httpd
423 ? S 0:00 httpd
424 ? SW 0:00 [httpd]
425 ? SW 0:00 [httpd]
426 ? SW 0:00 [httpd]
427 ? SW 0:00 [httpd]
428 ? SW 0:00 [httpd]
429 ? SW 0:00 [httpd]
430 ? SW 0:00 [httpd]
436 ? SW 0:00 [httpd]
437 ? SW 0:00 [httpd]
438 ? SW 0:00 [httpd]
470 ? S 0:02 xfs -port -1
485 ? SW 0:00 [smbd]
495 ? S 0:00 nmbd -D
533 ? SW 0:00 [postmaster]
538 tty1 SW 0:00 [mingetty]
539 tty2 SW 0:00 [mingetty]
540 tty3 SW 0:00 [mingetty]
541 tty4 SW 0:00 [mingetty]
542 tty5 SW 0:00 [mingetty]
543 tty6 SW 0:00 [mingetty]
544 ? SW 0:00 [prefdm]
549 ? SW 0:00 [prefdm]
559 ? S 0:02 [kwm]
585 ? S 0:06 kikbd
594 ? S 0:00 kwmsound
595 ? S 0:03 kpanel
596 ? S 0:02 kfm
597 ? S 0:00 krootwm
598 ? S 0:01 kbgndwm
611 ? S 0:00 kcmlaptop -daemon
666 ? S 0:00 /usr/libexec/postfix/master
668 ? S 0:00 qmgr -l -t fifo -u
787 ? S 0:00 pickup -l -t fifo
790 ? S 0:00 telnetd: 192.168.1.2 [vt100]
791 pts/0 S 0:00 login -- rich
792 pts/0 S 0:00 -bash
805 pts/0 R 0:00 ps ax
The first column in the output shows the process ID (or PID) of the process. Notice that the first process is our friend the init process, and assigned PID 1 by the Linux system. All other processes that start after the init process are assigned PIDs in numerical order. No two processes can have the same PID.
The third column shows the current status of the process (S for sleeping, SW for sleeping and waiting, and R for running). The process name is shown in the last column. Processes that are in brackets are processes that have been swapped out of memory to the disk swap space due to inactivity. You can see that some of the processes have been swapped out, but most of the running processes have not.
Still another responsibility for the kernel is hardware management. Any device that the Linux system must communicate with needs driver code inserted inside the kernel code. The driver code allows the kernel to pass data back and forth to the device, acting as a middle man between applications and the hardware. There are two methods used for inserting device driver code in the Linux kernel:
Drivers compiled in the kernel
- Driver modules added to the kernel
Previously, the only way to insert device driver code was to recompile the kernel. Each time you added a new device to the system, you had to recompile the kernel code. This process became even more inefficient as Linux kernels supported more hardware. Fortunately, Linux developers devised a better method to insert driver code into the running kernel.
Programmers developed the concept of kernel modules to allow you to insert driver code into a running kernel without having to recompile the kernel. Also, a kernel module could be removed from the kernel when the device was finished being used. This greatly simplified and expanded using hardware with Linux.
The Linux system identifies hardware devices as special files, called device files. There are three different classifications of device files:
Character device files are for devices that can only handle data one character at a time. Most types of modems and terminals are created as character files. Block files are for devices that can handle data in large blocks at a time, such as disk drives.
The network file types are used for devices that use packets to send and receive data. This includes network cards and a special loopback device that allows the Linux system to communicate with itself using common network programming protocols.
Linux creates special files, called nodes, for each device on the system. All communication with the device is performed through the device node. Each node has a unique number pair that identifies it to the Linux kernel. The number pair includes a major and a minor device number. Similar devices are grouped into the same major device number. The minor device number is used to identify a specific device within the major device group. This is an example of a few device files on a Linux server:
$ ls -al sda* ttyS*
brw-rw---- 1 root disk 8, 0 May 5 2006 sda
brw-rw---- 1 root disk 8, 1 May 5 2006 sda1
brw-rw---- 1 root disk 8, 10 May 5 2006 sda10
brw-rw---- 1 root disk 8, 11 May 5 2006 sda11
brw-rw---- 1 root disk 8, 12 May 5 2006 sda12
brw-rw---- 1 root disk 8, 13 May 5 2006 sda131
brw-rw---- 1 root disk 8, 14 May 5 2006 sda14
brw-rw---- 1 root disk 8, 15 May 5 2006 sda15
brw-rw---- 1 root disk 8, 2 May 5 2006 sda2
brw-rw---- 1 root disk 8, 3 May 5 2006 sda3
brw-rw---- 1 root disk 8, 4 May 5 2006 sda4
brw-rw---- 1 root disk 8, 5 May 5 2006 sda5
brw-rw---- 1 root disk 8, 6 May 5 2006 sda6
brw-rw---- 1 root disk 8, 7 May 5 2006 sda7
brw-rw---- 1 root disk 8, 8 May 5 2006 sda8
brw-rw---- 1 root disk 8, 9 May 5 2006 sda9
crw------- 1 root tty 4, 64 Jun 29 16:09 ttyS0
crw------- 1 root tty 4, 65 May 5 2006 ttyS1
crw------- 1 root tty 4, 66 May 5 2006 ttyS2
crw------- 1 root tty 4, 67 May 5 2006 ttyS3
Different Linux distributions handle devices using different device names. In this distribution, the sda device is the first SCSI hard drive, and the ttyS devices are the standard IBM PC COM ports. The listing shows all of the sda devices that were created on the sample Linux system. Not all are actually used, but they are created in case the administrator needs them. Similarly, the listing shows all of the ttyS devices created.
The fifth column is the major device node number. Notice that all of the sda devices have the same major device node, 8, while all of the ttyS devices use 4. The sixth column is the minor device node number. Each device within a major number has its own unique minor device node number.
The first column indicates the permissions for the device file. The first character of the permissions indicates the type of file. Notice that the SCSI hard drive files are all marked as block (b) device, while the COM port device files are marked as character (c) devices.
Unlike some other operating systems, the Linux kernel can support different types of filesystems to read and write data to and from hard drives. Besides having over a dozen filesystems of its own, Linux can read and write to and from filesystems used by other operating systems, such as Microsoft Windows. The kernel must be compiled with support for all types of filesystems that the system will use. Table 1 lists the standard filesystems that a Linux system can use to read and write data.
Any hard drive that a Linux server accesses must be formatted using one of the filesystem types listed in Table below.
The Linux kernel interfaces with each filesystem using the Virtual File System (VFS). This provides a standard interface for the kernel to communicate with any type of filesystem. VFS caches information in memory as each filesystem is mounted and used.
The GNU utilities
Besides having a kernel to control hardware devices, a computer operating system needs utilities to perform standard functions, such as controlling the
Linux system kernel, he had no system utilities to run on it. Fortunately for him, at the same time he was working, a group of people were working together on the Internet trying to develop a standard set of computer system utilities that mimicked the popular Unix operating system.
The GNU organization (GNU stands for GNU’s Not Unix) developed a complete set of Unix utilities, but had no kernel system to run them on. These utilities were developed under a software philosophy called open source software (OSS).
The concept of OSS allows programmers to develop software and then release it to the world with no licensing fees attached. Anyone can use the software, modify it, or incorporate it into his or her own system without having to pay a license fee. Uniting Linus’s Linux kernel with the GNU operating system utilities created a complete, functional, free operating system.
While the bundling of the Linux kernel and GNU utilities is often just called Linux, you will see some Linux purists on the Internet refer to it as the GNU/Linux system to give credit to the GNU organization for its contributions to the cause.
The core GNU utilities
The GNU project was mainly designed for Unix system administrators to have a Unix-like environment available. This focus resulted in the project porting many common Unix system command line utilities. The core bundle of utilities supplied for Linux systems is called the coreutils package.
The GNU coreutils package consists of three parts:
- Utilities for handling files
- Utilities for manipulating text
- Utilities for managing processes
These three main groups of utilities each contain several utility programs that are invaluable to the Linux system administrator and programmer. Here we covers each of the utilities contained in the GNU coreutils package in detail.
The GNU/Linux shell is a special interactive utility. It provides a way for users to start programs, manage files on the filesystem, and manage processes running on the Linux system. The core of the shell is the command prompt. The command prompt is the interactive part of the shell. It allows you to enter text commands, interprets the commands, then executes the commands in the kernel.
The shell contains a set of internal commands that you use to control things such as copying files, moving files, renaming files, displaying the programs currently running on the system, and stopping programs running on the system. Besides the internal commands, the shell also allows you to enter the name of a program at the command prompt. The shell passes the program name off to the kernel to start it.
There are quite a few Linux shells available to use on a Linux system. Different shells have different characteristics, some being more useful for creating scripts and some being more useful for managing processes. The default shell used in all Linux distributions is the bash shell. The bash shell was developed by the GNU project as a replacement for the standard Unix shell, called the Bourne shell (after its creator). The bash shell name is a play on this wording, referred to as the ‘‘Bourne again shell’’.
Besides the bash shell we will cover several other popular shells in this book. Table below lists the different shells we will examine.
Most Linux distributions include more than one shell, although usually they pick one of them to be the default. If your Linux distribution includes multiple shells, feel free to experiment with different shells and see which one fits your needs.
The Linux desktop environment
In the early days of Linux (the early 1990s) all that was available was a simple text interface to the Linux operating system. This text interface allowed administrators to start programs, control program operations, and move files around on the system.
With the popularity of Microsoft Windows, computer users expected more than the old text interface to work with. This spurred more development in the OSS community, and the Linux graphical desktops emerged.
Linux is famous for being able to do things in more than one way, and no place is this more relevant than in graphical desktops. There are a plethora of graphical desktops you can choose from in Linux. The following sections describe a few of the more popular ones.
The X Windows system
There are two basic elements that control your video environment — the video card in your PC and your monitor. To display fancy graphics on your computer, the Linux software needs to know how to talk to both of them. The X Windows software is the core element in presenting graphics.
The X Windows software is a low-level program that works directly with the video card and monitor in the PC, and controls how Linux applications can present fancy windows and graphics on your computer.
Linux isn’t the only operating system that uses X Windows; there are versions written for many different operating systems. In the Linux world, there are only two software packages that can implement it.
The XFree86 software package is the older of the two, and for a long time was the only X Windows package available for Linux. As its name implies, it’s a free open source version of the X Windows software.
Recently, a new package called X.org has come onto the Linux scene. It too provides an open source software implementation of the X Windows system. It is becoming increasingly popular, with many Linux distributions starting to use it instead of the older XFree86 system.
Both packages work the same way, controlling how Linux uses your video card to display content on your monitor. To do that, they have to be configured for your specific system. That is supposed to happen automatically when you install Linux.
When you first install a Linux distribution, it attempts to detect your video card and monitor and then creates an X Windows configuration file that contains the required information. During installation you may notice a time when the installation program scans your monitor for supported video modes. Sometimes this causes your monitor to go blank for a few seconds.Because there are lots of different types of video cards and monitors out there, this process can take a little while to complete.
This is where many of the customized Linux distributions can be lifesavers. Most of them take great effort to automatically detect video hardware and settings without asking you any technical questions. Unfortunately, sometimes the installation can’t autodetect what video settings to use, especially with some of the newer, more complicated video cards. Unfortunately, some Linux distributions will fail to install if they can’t find your specific video card settings. Others will ask a few questions during installation to help manually gather the necessary information. Still others default to the lowest common denominator and produce a screen image that is not customized for your video environment.
To complicate matters more, many PC users have fancy video cards, such as 3-D accelerator cards, so they can play high-resolution games. In the past, this caused a lot of problems if you tried to install Linux. But lately, video card companies are helping to solve this problem by providing Linux drivers. And many of the customized Linux distributions now include drivers for specialty video cards.
The core X Windows software produces a graphical display environment, but nothing else. While this is fine for running individual applications, it is not too useful for day-to-day computer use. There is no desktop environment allowing users to manipulate files or launch programs. To do that, you need a desktop environment on top of the X Windows system software.
The KDE desktop
The K Desktop Environment (KDE) was first released in 1996 as an open source project to produce a graphical desktop similar to the Microsoft Windows environment. The KDE desktop incorporates all of the features you are probably familiar with if you are a Windows user. Figure below shows a sample KDE desktop running on Linux.
The KDE desktop allows you to place both application and file icons on the desktop area. If you single-click an application icon, the Linux system starts the application. If you single-click on a file icon, the KDE desktop attempts to determine what application to start to handle the file. The bar at the bottom of the desktop is called the Panel. The Panel consists of four parts:
The K menu: Similarly to the Windows Start menu, the K menu contains links to start installed applications.
- Program shortcuts: These are quick links to start applications directly from the Panel.
- The taskbar: The taskbar shows icons for applications currently running on the desktop.
- Applets: These are small applications that have an icon in the Panel that often can change depending on information from the application.
All of the Panel features are Similar to what you would find in Windows. Besides the desktop features, the KDE project has produced a wide assortment of applications that run in the KDE environment. These applications are shown in Table below. (You may notice the trend of using a capital K in KDE application names.)
This is only a partial list of applications produced by the KDE project. There are lots more applications that are included with the KDE desktop.
The GNOME desktop
The GNU Network Object Model Environment (GNOME) is another popular Linux desktop environment. First released in 1999, GNOME has become the default desktop environment for many Linux distributions (the most popular being Red Hat Linux).
While GNOME chose to depart from the standard Microsoft Windows look-and-feel, it incorporates many features that most Windows users are comfortable with:
desktop area for icons
- Two panel areas
- Drag-and-drop capabilities
Figure below shows the standard GNOME desktop used in the Fedora Linux distribution
Figure: A GNOME desktop on a Fedora Linux system
Not to be outdone by KDE, the GNOME developers have also produced a host of graphical applications that integrate with the GNOME desktop. These are shown in Table below.
As you can see, there are also quite a few applications available for the GNOME desktop. Besides all of these applications, most Linux distributions that use the GNOME desktop also incorporate the KDE libraries, allowing you to run KDE applications on your GNOME desktop.
The downside to a graphical desktop environment is that they require a fair amount of system resources to operate properly. In the early days of Linux, a hallmark and selling feature of Linux was its ability to operate on older, less powerful PCs that the newer Microsoft desktop products couldn’t run on. However, with the popularity of KDE and GNOME desktops, this hallmark has changed, as it takes just as much memory to run a KDE or GNOME desktop as the latest Microsoft desktop environment.
If you have an older PC, don’t be discouraged. The Linux developers have banded together to take Linux back to its roots. They’ve created several low-memory-oriented graphical desktop applications that provide basic features that run perfectly fine on older PCs.
While these graphical desktops don’t have a plethora of applications designed around them, they still run many basic graphical applications that support features such as word processing, spreadsheets, databases, drawing, and, of course, multimedia support.
Table below shows some of the smaller Linux graphical desktop environments that can be used on lower-powered PCs and laptops.
These graphical desktop environments are not as fancy as the KDE and GNOME desktops, but they provide basic graphical functionality just fine. Figure below shows what the fluxbox desktop used in the SimplyMEPIS antiX distribution looks like. If you are using an older PC, try a Linux distribution that uses one of these desktops and see what happens. You may be pleasantly surprised.
Figure: The fluxbox desktop as seen in the SimplyMEPIS antiX distribution