Troubleshooting system resources (213.3)

Revision: $Revision$

Candidates should be able to identify, diagnose and repair local system issues when using software from the command line.

Key Knowledge Areas

Core system variables

The contents of:

/etc/profile && /etc/profile.d/






or other appropriate global shell configuration files

Any standard editor

Standard tools, utilites and commands to manipulate the above files and variables

The following is a partial list of the used files, terms and utilities:


Core system variables

Applications, for example bash, often use variables. These variables are not necessarily available as environment variables, but often have a local scope: they are available for the process, but not for its children. On the other hand, every environmental variable set by an application is available for every child of that application. For example: within a shell you can access environment variables as if they were locally defined shell-variables. You can promote a shell variable to an environment variable by using the export builtin.

Many environment settings influence the behaviour of applications. Some of them are very specific for just one application. But there are a number of standard variables that many applications will use or expect to be in place. A list of the most commonly used environment variables follows:

Table 13.1. Commonly used environment variables

COLUMNSthe number of columns your screen has (typically 80)
HOMEthe location of your home-directory
IFSInput File Separator. IFS contains the list of separators that the shell recognizes. By default this is set to whitespace values (space, tab etc.). However, setting IFS to a non-whitespace character can pose a security risk. For example: IFS=o, the program executes the command /bin/show. o is considered the separator. Hence /bin/sh would be executed. Most modern command shells therefore ignore the value for IFS.
LOGNAMEname used to log in
MANPATHcolon delimited sequence of directories that contain (sources for) manual pages, see manpath(1)
PAGERthe program used to display files on screen (e.g. less or pg)
PATHcolon delimited sequence of directories that contain executables
PS1..4definition of the prompts the shell uses in interactive mode: PS1: the primary prompt string, shown when you initially start typing a command; PS2: secondary prompt string, shown on the second and consecutive lines if you type in a command that spans multiple lines; PS3: the prompt used when bash issues the select command[a]; PS4: the value is printed before each command bash displays during an execution trace (e.g. set -x).
ROWSthe number of rows your screen has (typically 24)
SHELLthe name of the current shell
SHLVLif a shell is spawned, this variable is passed on to it. It will contain the SHLVL value of the parent shell, incremented by one.
TERMthe terminal type. Originally, this specified the type of terminal you physically had connected to the computer. Currently, virtual terminals are more often the case. This environment variable is used to select which controlling code sequences are sent to your (virtual) terminal, for example to set high intensity or clear the screen.
TMPDIRallows programs that use the tempnam(3) function call to use the directory specified by this variable rather than the /tmp directory.
UIDthe current users' id
USERthe current users name

[a] The select command is rarely used. It allows you to specify a number of options which are printed as a menu. You are then prompted with the PS3 prompt and can select an option, which is passed to a list of statements for further processing. See the bash manual page for more information.

To see your environment from within a shell, type env or set. Of course, the value of individual variables can be printed using the echo.


/etc/profile is the system wide profile for the bourne shell and bourne compatible shells. Below the /etc/profile file from a Debian system is displayed:

# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

if [ "`id -u`" -eq 0 ]; then
export PATH

if [ "$PS1" ]; then
  if [ "$BASH" ]; then
    # The file bash.bashrc already sets the default PS1.
    # PS1='\h:\w\$ '
    if [ -f /etc/bash.bashrc ]; then
      . /etc/bash.bashrc
    if [ "`id -u`" -eq 0 ]; then
      PS1='# '
      PS1='$ '

# The default umask is now handled by pam_umask.
# See pam_umask(8) and /etc/login.defs.

if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i
  unset i


The /etc/profile.d folder contains specific startup files for different shells. Any file in this folder which is readable will be executed by any shell started.

Below the contents of the file of a Debian system is displayed. This is the only file in the /etc/profile.d directory in a standard Debian system.

debian-601a:/etc/profile.d$ cat
# Check for interactive bash and that we haven't already been sourced.
[ -z "$BASH_VERSION" -o -z "$PS1" -o -n "$BASH_COMPLETION" ] && return

# Check for recent enough version of bash.
bash=${BASH_VERSION%.*}; bmajor=${bash%.*}; bminor=${bash#*.}
if [ $bmajor -gt 3 ] || [ $bmajor -eq 3 -a $bminor -ge 2 ]; then
    if [ -r /etc/bash_completion ]; then
        # Source completion code.
        . /etc/bash_completion
unset bash bmajor bminor


This is the systemwide bashrc file. This file is sourced in the /etc/profile file. So all bash shells started go through /etc/profile and /etc/bashrc.

Within the Debian distribution the /etc/bashrc is moved to /etc/bash.bashrc.

Login shells

The file /etc/shells lists the valid login shells on your system. It requires fully specified filenames. An example file is:


/bin/false is not a real shell, however, sometimes you want to add a user to your system that is not allowed to log in (e.g. wwwrun. Adding a user to your system requires giving him an entry in /etc/passwd, which in turn requires you to specify a valid login shell. Being able to specify /bin/false as shell prevents users to obtain a shell. The command chsh can be used by the user to set his preferred shell. root can use this command to set any user's shell. This command with the -l option will list the available shells. The -s option will allow the user to change his login shell:

$ chsh -s /bin/sh 

Shell startup environment

Most of your users log in and are presented with a shell. A shell's behaviour typically is controlled by setting a number of environmental variables. These are typically set at two levels: system wide and per user. To set them, a start-up file is loaded and executed. There are many shells - most of them have specific start-up files. Additionally, it depends on whether or not the shell is spawned by the login program or by another shell which startup files are executed. However, when shells are reported to display unexpected behaviour, often something is set wrongly in one of the start-up files.

Most login-shells will check for /etc/profile first. Typically, this file sets the TERM and PATH variables. The format for this file is the same as that of a (Posix) shell. Hence it contains calls to external programs, in older days, the fortune command was often called here: it generates a random epigram. On older systems the contents of the message of the day file (/etc/motd) were displayed here too, on newer systems this is often done by the login program itself.

More settings can be put in (often shell-specific) startup files in the users HOME directory. Posix compliant shells will check for $HOME/.profile. bash will behave likewise, if invoked as sh. Often, on Linux systems, /bin/sh is a (symbolic) link to /bin/bash. Some distributions put calls to distribution or shell specific central startup files into the user level files. This can be confusing at times.

bash is the most commonly used shell for Linux systems. It will run /etc/profile if invoked from login, then look for $HOME/.bash_profile, $HOME/.bash_login and $HOME/.profile, in that order. And then read and execute commands from the first one that exists and is readable. When a bash login shell exits, it reads and executes commands from the file $HOME/.bash_logout if readable. A common mistake is to create a userlevel start up file as root - and forget to set the permissions correctly. At least the user should have execute and read permissions. Typically the permissions should be -rwxr-x--- (assuming the file is owned by the user).

If users complain that their shell does not recognize certain key strokes or reacts strangely on them, chances are that either the TERM variable does not match the (virtual) terminal type in use or the key bindings for the application (often a shell) are wrongly set. If the TERM setting is invalid the display will often be garbled too.

The behaviour of programs that use the readline library to fetch input from the keyboard are influenced by key bind settings. The readline library supports emacs and vi key bindings by default and allows you to configure key bindings to certain functions within applications, for example bash. If the shell variable INPUTRC is set (often in /etc/profile) key-bindings are set by the file declared in the its value. Otherwise the key-bindings are set in the file $HOME/.inputrc for each individual user, or system wide in the file /etc/inputrc. The syntax for controlling key bindings is simple. All that is required is the name of the command or the text of a macro and a key sequence to which it should be bound, see the manual pages for readline(3). bash allows the current key bindings to be displayed or modified with the bind builtin command. The editing mode bash uses may be switched during interactive use by using the set -o option, e.g. set -o vi. Often, the user sets his preference in one of the startup files. A simple typo there can result in eccentric behaviour of the shell (at least for that user).

The file /etc/login.defs (man login.defs(5)) defines the behaviour of login on systems that use shadow passwords. With it, you control the behaviour of related programs too, for example chfn, chsh or groupadd. It is a plain text configuration file, used to set the initial PATH and other parameters, including how often a user must change passwords and what is acceptable as a password.

Many applications make use of other configuration files in the users HOME directory, for example:

Table 13.2. Commonly used configuration files in HOME

.emacsa configuration file for the emacs editor that contains ELISP functions
.exrc startup file for the vi editor
.fvwmrc startup file for the fvwm window manager
.ssh actually a directory, that contains files with the known hosts, identity and public key used by ssh
.twmrc another one, for the twm window manager
.newsrc keeps track to which newsgroups the user is subscribed and which articles he has read.
.Xdefaults behaviours, fonts, and colours for X Windows System applications
.xinitrc shell commands that run when the user logs into an X session
.xsession shell commands that run when the user logs into an X session using Xdm

It's easy for a user to overwrite (clobber) existing files when they are redirecting output to a file. To prevent this from happening to them you can set the noclobber setting by adding the line

set -o noclobber 

to their shell start-up file.


The /etc/init.d folder is discussed in the section called “/etc/init.d”.


The /etc/rc.xd folders contain the symbolic links to the runlevel scripts in folder /etc/init.d. For each runlevel there is a specific folder.

See /etc/init.d above for detailed description.


The Unix system engineer or system administrator can barely do his work without a proper editor. The definition of a proper editor however is, and always has been, discussed at length. Roughly, there are two mainstream editors available in the Unix world - though strictly speaking one of them is far more than just an editor - emacs and vi.

vi - or one of its clones, for example the popular vim by Bram Molenaar, used on most Linux installations - is a crude, but effective editor. It supports an interface that allows you to keep your fingers on the keyboard all the time, and by design makes you use as little keys as possible. E.g. the use of arrow keys, function keys and other non standard keys is non-mandatory. This results in a steep learning curve for novice users but they gain a high return on that investment. They will be able to process texts faster and will be able to maintain an ergonomically acceptable working position. They will be able to work on even slightly defective keyboards/terminals. vi has limited functionality, but everything one needs to effectively and efficiently edit texts is available.

On the other side of the spectrum resides emacs. Strictly speaking, it is a text processing system. It makes use of ELISP programs to enhance its functionality. emacs also supports an interface that allows you to keep your fingers on the keyboard all the time. The amount of features and extensions emacs offers can initially be somewhat overwhelming, but it comes with excellent built in documentation. Additionally a lot of people wrote extensions for emacs. For example evi, which emulates vi within emacs.

Both emacs and vi are widely used, however, vi is much smaller in size than the complete emacs package. Hence, and given its longer history, vi is available on almost every Unix platform (including Linux) and on every Unix system. I therefore strongly suggest that novices start using vi for a while. It is omnipresent and will fit on a boot disk or rescue disk, with plenty of room to spare. Check out emacs if you need features like spell-checking or syntax highlighting. There is no reason not to use both applications, but as a system administrator you must know the vi basics, especially if you have to work on older systems. Many systems do not have emacs installed, most Linux systems do, however.

Setting kernel parameters


sysctl.conf - sysctl(8) preload/configuration file

sysctl.conf is a simple file containing sysctl values to be read in and set by sysctl(8). The syntax is simply as follows:

              # comment
              ; comment

                token  =   value

Note that blank lines are ignored, and whitespace before and after a token or value is ignored, although a value can contain whitespace within. Lines which begin with a # or ; are considered comments and ignored.


sysctl - configure kernel parameters at runtime

       sysctl [-n] [-e] variable ...
       sysctl [-n] [-e] [-q] -w variable=value ...
       sysctl [-n] [-e] [-q] -p [filename]
       sysctl [-n] [-e] -a
       sysctl [-n] [-e] -A

sysctl is used to modify kernel parameters at runtime. The parameters available are those listed under /proc/sys/. Procfs is required for sysctl support in Linux. You can use sysctl to both read and write sysctl data.

To display all available configurable parameters:

$ sysctl -a

To read a specific kernel parameter:

$ sysctl kernel.msgmax
kernel.msgmax = 8192

To change a specific kernel parameter:

$ sudo sysctl kernel.msgmax=16384
[sudo] password for user:
kernel.msgmax = 16384

The sysctl command is used to modify kernel parameters at runtime. In Linux, it's a front end to the /proc filesystem, hence you should have compiled support for procfs if you want to use sysctl on Linux.

Typically you can read or set things like whether or not the kernel reacts to the three finger salute (ctrl+alt+del), the abilities of certain devices, such as your CDROM's ability to be able to close and open its tray, information about the way the kernel handles filesystem, such as the maximal number of inodes a filesystem can have, network information like whether or not the kernel should use IP forwarding, the kernel type and release and many, many more.

To see a list of all options available on your system you can issue the command:

# sysctl -a |more

You will see a listing that could start like this:

# sysctl -a |more
sunrpc.nlm_debug = 0
sunrpc.nfsd_debug = 0
sunrpc.nfs_debug = 0
sunrpc.rpc_debug = 0
dev.cdrom.check_media = 0
dev.cdrom.lock = 1
dev.cdrom.debug = 0
dev.cdrom.autoeject = 0
dev.cdrom.autoclose = 1 = CD-ROM information, Id: cdrom.c 3.11 2000/06/12 = = drive name:            hdc = drive speed:           24 = drive # of slots:      1 = Can close tray:        1 = Can open tray:         1 = Can lock tray:         1 = Can change speed:      1 = Can select disk:       0 = Can read multisession: 1 = Can read MCN:          1 = Reports media changed: 1

As you can see, the variable names (a.k.a. key names) consist of dot delimited parts. Each part of a variable name correlates to a directory under the /proc/sys hierarchy, for example /proc/sys/dev/cdrom/info corresponds to An alternate method to set or read the variables is by using plain old cat, e.g. you can either type:

# cat /proc/sys/dev/cdrom/info

.. or ..

# sysctl

Note, however, that sysctl will prepend the key name by default, use the -n option to suppress this. To confuse matters more, you are also allowed to use the / slash as a separator with sysctl, hence these two alternate forms are valid too:

# sysctl dev/cdrom/info
# sysctl /dev/cdrom/info

sysctl can load configuration files and set the kernel environment accordingly. The default location for the configuration file is /etc/sysctl.conf. Its syntax is simple: it contains simple token = value pairs, along with comment lines and/or blank lines. An example follows:

# sysctl.conf sample
kernel.domainname =
net.ipv4.ip_forward = 0

This example just sets the domain name and disables IP forwarding. You can use the -p option to instruct sysctl to read this file (or specify another one to read). Note, that it is possible to put whitespace and other (unmeant) tokens in the right hand side of the expressions in the configuration file. Also note that you can use the /proc interface to influence these variables, for example:

# echo "/sbin/modprobe" >/proc/sys/kernel/modprobe

Shared libraries

A library is an archive whose members are object files, i.e. pre-compiled program code. A library makes it possible for a program to use common routines without the administrative overhead of maintaining source code. Additionally the objects in the library do not have to be compiled each time the program is compiled.

There are two types of libraries: static libraries and shared or dynamic libraries.

The term static refers to the fact that, when a static library is used whilst creating a program, the linker links in the relevant parts of the library statically, i.e. they become part of the resulting executable file. This consumes unnecessary memory and disk space since each new instance of a statically linked program loads parts of the same program-code its predecessors did. The exact same code will be on disk and in memory many times. Also, bug-fixing code in a statically linked library requires recompilation of all programs that use the older version of the statically linked program.

To improve this shared libraries were invented. When a program uses a shared library, the binary just contains a reference to the library, not the code itself. A special run time loader, (for a.out format files) or (for ELF format files) finds the library at runtime and loads it into memory. Some additional naming conventions are used to allow realtime updates for shared libraries and support for non backward-compatible versions. Understanding these conventions is key to your ability to maintain a Linux system.

Shared object version numbering

By convention, a shared library has a major version number, a minor version number and a patch level identification (often also a number). The patch level is typically changed to specify that a bug fix or minor optimization of the code occurred: version 1.0.12 probably signifies the 12th bug-fix in version 1.0. Minor version numbers are typically incremented when a large number of patches have been submitted or a major bug was fixed. Major version numbers are changed when the interface to the functions in the shared library changes. Major versions of shared libraries should be backward compatible, hence programs that ran fine using a shared object version 1.0 should also run fine with version 1.1 or with version 1.9.9.

Naming schemes for shared objects

On a Linux system (and on many other Unix systems too) a shared object has a real name, a soname and a linkname. The linkname just contains the name of the library, the soname contains the major version number, and the real name contains the minor version and (when available) the patch level. On a Linux system, the soname and linkname are symbolic links to the correct shared library.

  • the real name is the name of the file that contains the shared objects precompiled code. By convention the format[.major][.minor][.patchlevel] is used: /lib/; the dynamic loader needs to know this name to find the exact instance of the shared library;

  • the soname often is the real name without the minor version number and the patch level number: /lib/ Programs that need a shared object will commonly use the soname to specify the library. On a Linux system that soname is a symbolic link to the real name of the shared library, hence the loader is able to find it;

  • the linkname is the soname without any versioning information: /lib/ The linkname of a shared object is needed as a reference to the shared object during the linking phase of a program (compile time). On a Linux system it is often a link to the soname (which in turn is a link to the real name).

The image below sketches a typical naming scheme for a shared object on a Linux system in this case for the foo library (i.e. version 1.0.4):

In this case, the compiler/linker and the executable (by means of the run-time loader) will use the exact same shared object, which indeed is the normal situation.

The symbolic link for the soname (which points to the real name) is created automatically when you run the ldconfig command (as root), which will be discussed in more detail in a later section. The linkname is typically set up during installation of a shared library and normally is a symlink to the soname or real name.

By pointing the symlink to another library, you can develop code using another library:

Now, you can run code using one version of a library (the soname symlink will be followed) and develop using another.

Another possibility is upgrading your shared object with a newer one. This typically is done like this:

So: /usr/lib/ is a (fully-qualified) soname, normally set by ldconfig to be a symbolic link to a real name like /usr/lib/ The linkname would be /usr/lib/, which typically is a symbolic link referring to /usr/lib/ too, but may be another name.


The ldconfig command performs two important functions: it creates a cache file, that speeds up loading of shared objects and creates the proper symbolic links to enable the linking loader to find the necessary shared object. For a better understanding of how ldconfig achieves this, you need to know a bit more about the file format(s) of shared objects.

A shared object is recognizable by its file format. On Intel/Linux that typically is an ELF file format, though the conventional a.out format is still supported on some systems. The conventional a.out format will not be discussed here in much detail, since on most Unix/Linux systems the ELF file format is standard nowadays.

There are two main stream types of ELF libraries: those linked against libc5 and those linked against libc6 (glibc). Also, there can be shared objects that need both.

An ELF file contains a header and various sections; the information contained in the sections depends on the type of ELF file. A shared object however contains a Dynamic Section (DS), which in turn contains a number of symbols and corresponding values. One of these symbols is named SONAME and its value contains the soname as specified by the programmer. ldconfig uses this value to determine the SONAME for a file that it has detected to be a valid shared object. Also, the files filename is considered.

Note, that the SONAME in the DS is not required to match the (an) actual filename for that shared object e.g., the shared object could be in a file named meaninglessfilename while the SONAME in the DS could be

If you want to inspect the SONAME in the DS, you can use the objdump utility:

$ objdump -p /usr/lib/ |grep SONAME

ldconfig uses both the SONAME as stored in the shared object and the filename it has to create the cache file and the appropriate symbolic links. ldconfig scans all directories whose names are passed to it on the command line and in the file /etc/ The two trusted directories /usr/lib and /lib are always scanned too.

For each directory specified as location for shared objects ldconfig scans the files found there. It skips those files it does not recognize as a shared object. Shared objects are recognized as such if they are either in a.out format or in ELF format.

For each shared object its SONAME will be determined next. An internal list is built that contains all SONAMES and their corresponding filenames. Most actual is determined by the shared library's filename, adhering to the conventions stated before, hence is regarded as more recent than

If all shared objects for a given directory have been scanned the appropriate symbolic links will be made: for each SONAME in the list a symbolic link to the most recent version of the shared object (according to its version number) will be created. Next, the cache file will be written, consisting of the SONAMES, type of library and accompanying full file name.

How the dynamic linker locates shared objects

As you know, major versions of shared libraries are backward compatible. Therefore, most programs suffice by specifying the major version number to the dynamic loader (the soname).

  • the linker for ELF format files,, starts by checking the environment variable LD_LIBRARY_PATH. This variable may contain a colon separated list of directories to search. Often used to permit non-'root' users to define the location for self-written shared objects, since a non-'root' user is not permitted to put these in the standard locations, or to allow testing;

  • if the library could not be found in the directories that are mentioned in the environment variable, the cache file is searched. That file, by default /etc/, contains a compiled list of libraries and typically is created using ldconfig as we described earlier. The file /etc/ is not human readable. If you want to see the contents of the cache file use the command ldconfig -p /etc/

    # ldconfig -p /etc/
    432 libs found in cache `/etc/'
   (libc6) => /usr/lib/
   (libc6) => /usr/lib/
    ...deleted a lot of lines...
   (ELF) => /lib/
   (ELF) => /lib/

  • Shared libraries were traditionally stored in /lib and /usr/lib. These locations are searched as a last resort. This will probably never happen unless the cache file does not exist because ldconfig has already scanned those directories when creating the cache file.

To see which versions of which shared objects are needed by a program, use the command ldd. This command also tries to resolve the matching filename, it uses the cache file to do this. For example:

# ldd /usr/bin/vim => /lib/ (0x4001a000) => /lib/ (0x40063000)
        /lib/ => /lib/ (0x40000000)
# _

As you see, the author of vim choose to use the ncurses shared library - to be precise: the library with the SONAME, which translates to the fully qualified filename /lib/

There are a number of occasions in which the exact location of shared libraries needs to be known. When a program is linked, the linker (ld) needs their location to be able to refer to them. It searches for them in /lib, /usr/lib and in directories added with -rpath and -Ldir command line options.

gcc -shared -Wl,-soname, -o jvc.o
ln -sf
ln -sf
ln -sf

Copyright Snow B.V. The Netherlands