Troubleshooting environment configurations (213.4)

Revision: $Revision$

Candidates should be able to identify common local system and user environment configuration issues and common repair techniques.

Key Knowledge Areas

Core system variables

init configuration files

init start process

cron configuration files

Login process

User-password storage files

Determine user group associations

SHELL configuration files of bash

Analysing which processes or daemons are running

The following is a partial list of the used files, terms and utilities:

/etc/inittab
/etc/rc.local
/etc/rc.boot
/var/spool/cron/crontabs/
The default shell configuration file(s) in /etc/
/etc/login.defs
/etc/syslog.conf
/etc/passwd
/etc/shadow
/etc/group
/sbin/init
/etc/profile
/usr/sbin/cron
/usr/bin/crontab

Core system variables

Core system variables are defined during the login process. During the login proces a number of files are passed. Commands in those file are executed, before a user shell is started. Files and directories that are read during the login proces are: /etc/profile && /etc/profile.d/ and /etc/bashrc. A lot of information about these files and directories can be found in the previous chapter 2.13.3. See the section called “Core system variables”

Things to know about /etc/profile and bashrc:

/etc/profile is executed only when a new login shell is started.

Commands in /etc/profile are executed at login time.

.bashrc in the user's directory is called for each new shell started.

Commands in /etc/bashrc are executed for each invocation of bash.

/etc/skel - new user directories are populated automatically by copying /etc/skel and its contents.

Login process

As described above during the login proces a number of files are read before a shell is started.

The default shell configuration file(s) in /etc/

/etc/profile && /etc/profile.d/and /etc/bashrc

init configuration files

During startup a number of files are read. The init configuration file is /etc/inittab. /etc/init.d is the main directory for startup scripts. These scripts have symbolic links to /etc/rcN.d. These scripts are started step by step. After the /etc/rcN.d scripts are run /etc/rc.local is the last file to be executed. The rc.local file can be configured for starting up local stuff.

/etc/inittab

inittab - format of the inittab file used by the sysv-compatible init process

The inittab file describes which processes are started at bootup and during normal operation (e.g. /etc/init.d/boot, /etc/init.d/rc, gettys...). Init(8) distinguishes multiple runlevels, each of which can have its own set of processes that are started. Valid runlevels are 0-6 plus A, B, and C for ondemand entries. An entry in the inittab file has the following format:

id:runlevels:action:process

Please refer to man inittab for explanation of the various inittab fields.

And check explanation of action files in the manpage of inittab.

Example /etc/inittab

              # Level to run in
              id:2:initdefault:

              # Boot-time system configuration/initialization script.
              si::sysinit:/etc/rc.sysinit

              # What to do in single-user mode.
              ~:S:wait:/sbin/sulogin

              # /etc/init.d executes the S and K scripts upon change
              # of runlevel.
              #
              # Runlevel 0 is halt.
              # Runlevel 1 is single-user.
              # Runlevels 2-5 are multi-user.
              # Runlevel 6 is reboot.

              l0:0:wait:/etc/rc 0
              l1:1:wait:/etc/rc 1
              l2:2:wait:/etc/rc 2
              l3:3:wait:/etc/rc 3
              l4:4:wait:/etc/rc 4
              l5:5:wait:/etc/rc 5
              l6:6:wait:/etc/rc 6

              # What to do at the "3 finger salute".
              ca::ctrlaltdel:/sbin/shutdown -t3 -r now

              # Runlevel 2,3: getty on virtual consoles
              # Runlevel   3: mgetty on terminal (ttyS0) and modem (ttyS1)
              1:23:respawn:/sbin/mingetty tty1
              2:23:respawn:/sbin/mingetty tty2
              3:23:respawn:/sbin/mingetty tty3
              4:23:respawn:/sbin/mingetty tty4
              S0:3:respawn:/sbin/agetty ttyS0 9600 vt100-nav
              S1:3:respawn:/sbin/mgetty -x0 -D ttyS1

/etc/rc.local

Originally on BSD systems /etc/rc.local was used for all local services. /etc/rc.local is almost the last script called at boot up. This script can be edited by the administrator to start local daemons. On a Debian system /etc/rc.local is installed as an empty file by default. This file can be used to start local services after everything else is started already. Below is the Debian /etc/rc.local file.

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

exit 0

/etc/rc.boot

/etc/rc.boot used to be a directory for local or per-package boot scripts. Currently the /etc/rc.boot directory is obsolete. It has been superseded by the /etc/rcS.d directory. At boot time, first the /etc/rcS.d directory is scanned and then, for backwards compatibility, the /etc/rc.boot directory.

Troubleshooting /etc/inittab and /sbin/init

The program /sbin/init reads the file /etc/inittab. See the section called “ What happens next, what does /sbin/init do? ” for a detailed description of the boot process.

If a process isn't running after booting the system, check this file for errors. Then find out what the default runlevel is and check the /etc/rc?.d/ directory for start/stop scripts.

To get an idea what kind of problems can occur if something is not configured correctly in /etc/inittab, have a look at the file and you'll see things like:

  • The default runlevel. This is the runlevel the system will be in when the boot stage is completed.

  • Scripts to run at boottime. These are the scripts in the directory /etc/rcS.d.

  • What to do if CTRL+ALT+DEL is pressed. So, if you don't want to have the system respond with a reboot to this famous key combination, you can change this behavior here.

  • The number of terminals (Alt+F1...Alt+Fn) that can be invoked by pressing one of the aforementioned key combinations.

  • Getty's on modem lines. A well known message is one of the form: Id ttyS3 respawning too fast: disabled for 5 minutes.

    The Modem-HOWTO on www.linuxdoc.org describes this as follows:

    The line mentioned, in this case ttyS3, causes the problem. Make sure the syntax for this line is correct and that the device (ttyS3) exists and can be found. If the modem has negated CD (Carrier Detect) and getty opens the port, you'll get this error message since negated CD will kill getty. Then getty will respawn only to be killed again, etc. Thus it respawns over and over (too fast). It seems that if the cable to the modem is disconnected or you have the wrong serial port, it's just like CD is negated. All this can occur when your modem is chatting with getty. Make sure your modem is configured correctly. Look at AT commands E and Q.

Bypassing /sbin/init in case of trouble

In case of trouble, when booting a system, init can be bypassed by entering init=/bin/bash in GRUB or LILO when starting the system. So a bash shell is started instead of /sbin/init. Now the system can be investigated.

cron configuration files

/var/spool/cron/crontabs

The directory /var/spool/cron/crontabs contains the individual crontabs.

This directory contains crontab(1) files for the adm, root, and sys logins. Users whose login names are in the /etc/cron.d/cron.allow file can establish their own crontab file using the crontab command. If the cron.allow file does not exist, the /etc/cron.d/cron.deny file is checked to determine if the user is denied the use of the crontab command.

As root, you can use the crontab command to make the desired entries. Revisions to the file take effect at the next reboot. The file entries support the calendar(1) reminder service and the Basic Networking Utilities (BNU). Remember, you can use the cron(1M) function to decrease the number of tasks you perform; include recurring and habitual tasks in your crontab file. See crontab(1).

Do not manually edit the crontab entries. Use crontab -e instead. When editing crontab make sure that you have the correct user id, before editing the crontab . Check it with the id command.

/usr/sbin/cron

cron - daemon to execute scheduled commands (Vixie Cron)

cron [-f] [-l] [-L loglevel]

cron is started automatically from /etc/init.d on entering multi-user runlevels.

OPTIONS
       -f      Stay in foreground mode, don't daemonize.

       -l      Enable LSB compliant names for /etc/cron.d files

       -L loglevel
               Sets the loglevel for cron. The standard logging level (1)  will  log
               the start of all the cron jobs. A higher loglevel (2) will cause cron
               to log also the end of all cronjobs, which can be useful to audit the
               behaviour  of  tasks  run  by  cron.  Logging will be disabled if the
               loglevel is set to zero (0).
       cron searches its spool area  (/var/spool/cron/crontabs)  for  crontab  files
       (which  are  named  after accounts in /etc/passwd); crontabs found are loaded
       into memory.  Note that crontabs in this directory  should  not  be  accessed
       directly - the crontab command should be used to access and update them.

Read the man page to determine in which order the different crontab files and crontab directories are read.

/usr/bin/crontab

crontab - maintain crontab files for individual users (Vixie Cron)

SYNOPSIS
       crontab [ -u user ] file
       crontab [ -u user ] [ -i ] { -e | -l | -r }

Crontab is the program used to install, deinstall or list the tables used to drive the cron(8) daemon in Vixie Cron. Each user can have their own crontab, and though these are files in /var/spool/cron/crontabs, they are not intended to be edited directly.

CRONTAB(5)

crontab - tables for driving cron

Commands are executed by cron(8) when the minute, hour, and month of year fields match the current time, and when at least one of the two day fields (day of month, or day of week) match the current time (see ``Note'' below). cron(8) examines cron entries once every minute. The time and date fields are:

              field          allowed values
              -----          --------------
              minute         0-59
              hour           0-23
              day of month   1-31
              month          1-12 (or names, see below)
              day of week    0-7 (0 or 7 is Sun, or use names)

       A field may be an asterisk (*), which always stands for ``first-last''.
/etc/cron.allow and/etc/cron.deny

If the /etc/cron.allow file exists, then you must be listed (one user per line) therein in order to be allowed to use this command. If the /etc/cron.allow file does not exist but the /etc/cron.deny file does exist, then you must not be listed in the /etc/cron.deny file in order to use this command.

If neither of these files exists, then depending on site-dependent configuration parameters, only the super user will be allowed to use this command, or all users will be able to use this command.

If both files exist then /etc/cron.allow takes precedence. Which means that /etc/cron.deny is not considered and your user must be listed in /etc/cron.allow in order to be able to use the crontab.

Troubleshooting cron processes

Cron is responsible for the running of scheduled processes. Each individual user can have his own crontab. If a user can't get his scheduled job to run, the most common causes for this are that the user lacks the authorization to run the command in question or the user lacks the authorization to use cron. The latter is the case if a file /etc/cron.allow exists and the user is not mentioned in that file or if the file /etc/cron.deny exists and the user is mentioned in that file.

The files /var/spool/cron/crontabs/<username> which are the user specific cron files, should not be edited directly, use crontab -e instead.

User-password storage files

/etc/login.defs the site-specific configuration for the shadow passwd file.

The shadow file for /etc/passwd is /etc/shadow

The shadow file for /etc/group is /etc/gshadow

/etc/login.defs

login.defs - shadow password suite configuration

The /etc/login.defs file defines the site-specific configuration for the shadow password suite. This file is required. Absence of this file will not prevent system operation, but will probably result in undesirable operation.

This file is a readable text file, each line of the file describing one configuration parameter. The lines consist of a configuration name and value, separated by whitespace. Blank lines and comment lines are ignored. Comments are introduced with a "#" pound sign and the pound sign must be the first non-white character of the line.

See man page login.defs for more information about this file. Look at /etc/login.defs. Many parameters are pre-defined. The standard /etc/login.defs file on a Debian/Ubuntu system contains a lot of comments.

Troubleshooting /etc/login.defs

This has already been described in the section called “Shell startup environment”.

/etc/passwd

passwd - the password file

/etc/passwd contains one line for each user account, with seven fields delimited by colons (:). These fields are:

       ·   login name

       ·   optional encrypted password

       ·   numerical user ID

       ·   numerical group ID

       ·   user name or comment field

       ·   user home directory

       ·   optional user command interpreter

First entries from a Debian /etc/passwd file:

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync

And /etc/passwd permissions:

$ ls -l passwd
-rw-r--r-- 1 root root 2141 2011-04-29 10:18 passwd
/etc/shadow

shadow - shadowed password file

shadow is a file which contains the password information for the systems accounts and optional aging information.

This file must not be readable by regular users if password security is to be maintained.

Each line of this file contains 9 fields, separated by colons (:), in the following order:

       login name
       encrypted password
       date of last password change
       minimum password age
       maximum password age
       password warning period
       password inactivity period
       account expiration date

First entries in the /etc/shadow file:

root:!:15039:0:99999:7:::
daemon:*:15039:0:99999:7:::
bin:*:15039:0:99999:7:::
sys:*:15039:0:99999:7:::
sync:*:15039:0:99999:7:::

And /etc/shadow permissions:

$ ls -l shadow
-rw-r----- 1 root shadow 1679 2011-04-29 10:18 shadow
/etc/group

group - user group file

/etc/group is a text file which defines the groups on the system. There is one entry per line, with the following format:

group_name:password:GID:user_list

The field descriptions are:

       group_name
              the name of the group.

       password
              the (encrypted) group password.  If this field is empty, no  password  is
              needed.

       GID    the numerical group ID.

       user_list
              a list of the usernames that are members of this group, separated by comma's.

First lines of /etc/group

root:x:0:
daemon:x:1:
bin:x:2:
sys:x:3:
adm:x:4:user

And /etc/group permissions:

$ ls -l group*
-rw-r--r-- 1 root root 1036 2011-04-12 16:29 group
/etc/gshadow

gshadow - shadowed group file

/etc/gshadow contains the shadowed information for group accounts.

This file must not be readable by regular users if password security is to be maintained.

Contents /etc/gshadow file first 5 entries:

root:*::
daemon:*::
bin:*::
sys:*::
adm:*::user

And /etc/gshadow permissions:

$ ls -l /etc/gshadow
-rw-r----- 1 root shadow 753 2011-04-12 16:29 /etc/gshadow
pwconv

pwconv, pwunconv, grpconv, grpunconv - convert to and from shadow passwords and groups

Each program acquires the necessary locks before conversion. pwconv and grpconv are similar. First, entries in the shadowed file which do not exist in the main file are removed. Then, shadowed entries which do not have 'x' as the password in the main file are updated. Any missing shadowed entries are added. Finally, passwords in the main file are replaced with 'x'. These programs can be used for initial conversion as well to update the shadowed file if the main file is edited by hand.

Commands to manipulate passwd, shadow, group and gshadow file
useradd user - create the account user
usermod user - modify user account
userdel user - delete the user account
groupadd group - add group
groupmod group - modify the parameters of group
groupdel group - delete group
passwd username - set/change the password for username
gpasswd groupname - set/change the passwd for groupname
pwconv - convert standard password file to shadow configuration
pwunconv - revert from a shadow passwd configuration
grpconv - convert a standard group file to a shadow configuration
grpunconv - revert from a shadow group configuration
chage user - modify password aging and expiration settings for user

Troubleshooting authorisation problems

The files involved are /etc/passwd, /etc/shadow and /etc/group.

If a user can't login or gets the wrong shell, chances are the problem is located in /etc/passwd or /etc/shadow. These kind of problems often occurs if someone has been editing the files manually instead of using one of the system tools such as adduser, useradd, deluser, userdel.

Suppose, for example, that a user gets the following result when issuing the command ls -l:

home# ls -l
total 8
dr-xr-xr-x    6 root     0            1024 Dec  6 19:07 ftp
drwxr-xr-x    6 www-data 33           1024 Oct  8 17:18 omproject
drwxr-xr-x    5 piet     1003         1024 Dec  6 23:34 piet
drwxr-xr-x    3 rc564    1001         1024 Dec  4 19:37 rc564
drwxr-xr-x    2 secofr   400          1024 Dec 17 15:26 secofr
drwxr-sr-x   38 willem   1000         3072 Feb  5 09:48 willem
      

Obviously, something has gone wrong: there are group numbers in the result instead of group names as it should be:

dr-xr-xr-x    6 root     root         1024 Dec  6 19:07 ftp
drwxr-xr-x    6 www-data www-data     1024 Oct  8 17:18 omproject
drwxr-xr-x    5 piet     piet         1024 Dec  6 23:34 piet
drwxr-xr-x    3 rc564    rc564        1024 Dec  4 19:37 rc564
drwxr-xr-x    2 secofr   secofr       1024 Dec 17 15:26 secofr
drwxr-sr-x   38 willem   willem       3072 Feb  5 09:48 willem
      

The problem lies in /etc/group which is used to replace the group number by the group name. The group is probably not defined in /etc/group. If /etc/passwd is edited manually, pwconv can be used to synchronize /etc/shadow.

Troubleshooting /etc/profile

Here you'll often find the system-wide variable definitions such as PATH, umask and the prompt. Typical problems that can be the result of an error in /etc/profile can be commands that can't be found (due to a wrong PATH) and new files that are created with the wrong permissions (due to an erroneous umask).

/etc/syslog.conf

Some distributions do not use syslog anymore, instead they use rsyslog. (Debian)

Below syslog.conf is explained.

syslog.conf - syslogd(8) configuration file

The syslog.conf file is the main configuration file for the syslogd(8) which logs system messages on *nix systems. This file specifies rules for logging. For special features see the sysklogd(8) manpage.

Every rule consists of two fields, a selector field and an action field. These two fields are separated by one or more spaces or tabs. The selector field specifies a pattern of facilities and priorities belonging to the specified action.

Lines starting with a hash mark (''#'') and empty lines are ignored.

syslogd

syslogd provides the kind of logging that is used by many modern programs. Every logged message contains at least a time, a hostname and, usually, a program name field. (See the syslog(3) manual page for information on generating messages)

Rules in /etc/syslog.conf specify where messages should go. An example syslog.conf entry is shown below:

	    # Sample syslog.conf
	    daemon.warning      /var/log/daemon.log
	  

In this example messages of priority warning and higher, from processes that use the facility daemon, will be sent to the file /var/log/daemon.log.

The new scheme adds four new specifiers. The asterisk (*) wildcard, the equation sign (=), the exclamation mark (!), and the minus sign (-).

The * specifies that all messages for the specified facility are to be directed to the destination. Note that this behaviour is degenerate with specifying a priority level of debug. Users have indicated that the asterisk notation is more intuitive.

The = wildcard is used to restrict logging to the specified priority class. This allows, for example, routing only debug messages to a particular logging source. For example the following line in syslog.conf would direct debug messages from all sources to the /usr/adm/debug file.

                   # Sample syslog.conf
                   *.=debug            /usr/adm/debug
		

The ! is used to exclude logging of the specified priorities. This affects all (!) possibilities of specifying priorities. For example the following lines would log all messages of the facility mail except those with the priority info to the /usr/adm/mail file. And all messages from news.info (including) to news.crit (excluding) would be logged to the /usr/adm/news file.

                # Sample syslog.conf
                mail.*;mail.!=info       /usr/adm/mail
                news.info;news.!crit     /usr/adm/news
		

You may use it intuitively as an exception specifier. The above mentioned interpretation is simply inverted. Doing that you may use:

                mail.none

or

                mail.!*

or

                mail.!debug

to skip every message that comes with a mail facility.

The - may only be used to prefix a filename if you want to omit sync'ing the file after every write to it. This may take some acclimatization for those individuals used to the pure BSD behaviour but testers have indicated that this syntax is somewhat more flexible than the BSD behaviour. Note that these changes should not affect standard syslog.conf(5) files. You must specifically modify the configuration files to obtain the enhanced behaviour.

The facility specifies the subsystem that produces the message. The following list contains all possible syslog facilities.

auth

Security and authorization messages. This facility is deprecated, use authpriv instead.

authpriv

Security and authorization messages.

cron

Clock daemons (at and cron).

daemon

Miscellaneous daemons without a separate facility value.

ftp

Ftp daemons.

kern

Kernel messages.

lpr

Line printer subsystem.

mail

Mail subsystem, including services like fetchmail and IMAP.

mark

Syslogd generated messages that contains a '- -MARK- -' string and a timestamp.

news

USENET news subsystem.

syslog

Messages generated internally by syslogd.

user

Generic user-level messages. This is the default facility.

uucp

UUCP subsystem.

local0 through local7

Reserved for local use.

The priority is one of the following keywords, in ascending order: debug, info, notice, warning, err, crit, alert, emerg. The priority defines the severity of the message. The standard behaviour is that all messages of the specified priority and higher are logged to the given action.

The second field is the logfile, but it need not be a regular file. The syslogd provides the following actions:

Regular File

Typically messages are logged to real files. The file has to be specified with the full pathname, beginning with a slash ('/').

You may prefix an entry with the minus ('-') sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right after a write attempt. Nevertheless, this might give you back some performance, especially if you run programs that use logging in a very verbose manner.

Terminal and Console

If the file you specified is a tty, special tty-handling is done, same as with /dev/console.

Remote Machine

syslogd provides full remote logging, that is, it is able to send messages to a remote host running syslogd and to receive messages from remote hosts. The remote host won't forward the message again, it will just log them locally. To forward messages to another host, prepend the hostname with the at sign ('@'):

# Send all logging to a remote machine
*.*   @remote

Using this feature, you're able to control all syslog messages on one host, assuming all other machines log remotely to that host. This makes administration easier.

If the remote hostname cannot be resolved at startup because the name-server is inaccessible (perhaps it is started after syslogd), no need to worry: syslogd will retry ten times before complaining. One way to avoid this is to place the hostname in /etc/hosts.

Note that you have to start the central logging server with the -r option to enable this. By default, syslogd doesn't listen to the network. Also note that syslogd has no facility to split log messages from different hosts to different files.

List Of Users

Usually, critical messages are also directed to root on the host machine. You can specify a list of users who get the message by simply writing their login names. You may specify more than one user by separating them with commas (',').

# Messages of the priority alert will be directed to root and joey
*.alert     root,joey

If they're logged in, they get the message. An e-mail will not be sent, because, by the time it arrives, it might be too late.

Emergency messages often should go to all users currently online to notify them that something strange is happening with the system. To specify this wall-feature, use an asterisk ('*').

Copyright Snow B.V. The Netherlands