Advanced Network Configuration and Troubleshooting (205.2)

The candidate should be able to configure a network device to implement various network authentication schemes. This objective includes configuring a multi-homed network device, configuring a virtual private network and resolving networking and communication problems.

Resources: Bird01, Drake00,

Key Knowledge Areas

  • Utilities to manipulate routing tables

  • Utilities to configure and manipulate ethernet network interfaces

  • Utilities to analyse the status of the network devices

  • Utilities to monitor and analyse the TCP/IP traffic

Terms and Utilities

  • /sbin/route

  • /sbin/ifconfig

  • /bin/netstat

  • /bin/ping

  • /usr/sbin/arp

  • /usr/sbin/tcpdump

  • /usr/sbin/lsof

  • /usr/bin/nc

  • /sbin/ip

  • nmap

  • wireshark

Virtual Private Network

What Is A VPN

A VPN (Virtual Private Network) allows you to connect two or more remote networks securely over an insecure connection, for example over the public Internet. To do this an encrypted secure tunnel is created: all data will be encrypted before being sent over the insecure network. The resulting network connection acts and feels like a physical connection, but actually may traverse many physical networks and systems. Hence its name: "virtual".

VPNs are frequently used to save costs. In olden days physical connections had to be leased from telecom providers or you had to use POTS or ISDN lines. This was a costly business. Nowadays the Internet is omnipresent, and almost always available at a low fixed monthly price. However, the Internet can be sniffed and intruders might inspect and/or intercept your traffic. A VPN shields you from most of the problems you might have otherwise.

A use case might be to integrate LANs in several offices or branches. A user that works in the Los Angeles office hence can access the network services of the department in New York vice versa. In most cases, offices already have an Internet connection, so no additional investments need to be made.

VPN Types

There are many ways to implement a VPN. Many companies use proprietary software implementations. Also many routers have VPN support built in. A usable VPN can be build using a simple SSH tunnel or by using a more sophisticated dedicated solution. Some VPN implementations include:

  • IPSEC

  • VPND

  • SSH

  • Many Cisco Routers (or other proprietary implementations)

  • SSL/TLS

This book will outline the implementations of OpenVPN and IPSEC. OpenVPNOpenVPN is covered separately in the chapter on System Security.

IPSEC

IPSEC provides encryption and authentication services at the IP (Internet Protocol) level of the network protocol stack. It replaces/enhances the regular level 3 IP layer so all packets are encrypted, including for example UDP packets. The IPSEC layer has been standardized by the IETF in RFCs 2401–2412. Use of IPSEC is an option for IPv4 but its use is mandatory in IPv6 stacks.

In a regular IPv4 network you might set up dedicated IPSEC gateway machines to provide encrypted IP network connections when needed. IPSEC can run on routers, firewall machines, various application servers and on end-user desktop or laptop machines - any system that has an IP stack.

Using IPSEC is simple, as the protocol is built-in into the IP stack. But there are additional tasks in comparison with a regular IPv4 connection, for example encryption keys need to be exchanged between the end-points before an encrypted tunnel can be set up. It was decided to handle this over a higher-level protocol, the Internet Key Exchange protocol (IKE). After IKE has done its work, the IP level services ESP and AH know which keys to use to do their work.

The full names of the three protocols that are used in an IPSEC implementation are:

ESP, Encapsulating Security Payload

Encrypts and/or authenticates data;

AH, Authentication Header

Provides a packet authentication service;

IKE, Internet Key Exchange

Negotiates connection parameters, including keys, for the protocols mentioned above. The IKE protocol ensures authentication of the peers and exchange of their symmetric keys. The IKE protocol is usually implemented by a user space daemon that uses port 500/udp.

Note

IPSEC standards define all three protocols, but in some contexts people use the term IPSEC to refer to just AH and ESP.

OpenSwan, formerly known as FreeS/WAN, is a complete IPSEC implementation for Linux 2.0 - 2.6 kernels. StrongSwan (also derived from FreeS/WAN) is another implementation that also supports the 3.x kernel. Both OpenSwan and FreeSwan implement all three protocols mentioned earlier. The Openswan implementation has several main parts:

  • KLIPS (KerneL IPSec) which implements generic IPSEC packet handling, AH and ESP on the kernel level, for all kernels before version 2.5.47. KLIPS has been superseded by native IPSEC kernel support (NETKEY).

  • NETKEY is the Kernel IPSec implementation included with the 2.6 kernel.

  • Pluto (an IKE daemon) implements IKE, negotiating connections with other systems.

  • various scripts provide an administrator interface to the machinery.

The config file contains three parts:

one or more connection specifications

Each connection section starts with conn ident, where ident is an arbitrary name which is used to identify the connection.

connection defaults

This section starts with conn %default. For each parameter in it, any section which does not have a parameter of the same name gets a copy of the one from the %default section. There may be multiple %default sections, but only one default may be supplied for any specific parameter name and all %default sections must precede all non-%default sections of that type.

the config section

The config section starts with config setup and contains information used when starting the software.

A sample configuration file is shown below:

	# basic configuration
	config setup
	        interfaces="ipsec0=eth0"
	        klipsdebug=none
	        plutodebug=none
	        plutoload=%search
	        plutostart=%search
	        # Close down old connection when new one using same ID shows up.
	        uniqueids=yes

	# defaults that apply to all connection descriptions
	conn %default
	        # How persistent to be in (re)keying negotiations (0 means very).
	        keyingtries=0
	        # How to authenticate gateways
	        authby=rsasig

	# VPN connection for head office and branch office
	conn head-branch
	        # identity we use in authentication exchanges
	        leftid=@head.example.com
	        leftrsasigkey=0x175cffc641f...
	        left=45.66.9.170
	        leftnexthop=45.66.11.254
	        leftsubnet=192.168.11.0/24
	        # right s.g., subnet behind it, and next hop to reach left
	        rightid=@branch.example.com
	        rightrsasigkey=0xfc641fd6d9a24...
	        right=17.120.138.134
	        rightnexthop=17.120.138.1
	        rightsubnet=192.168.0.0/24
	        # start automatically when ipsec is loaded
	        auto=start
					

In a typical setup you have two interconnected gateways that both run IPSEC and route packets. One of these gateways can be seen as 'the one on the left', the other as 'the one on the right'. Hence specifications are written in terms of left and right participants. There is no special meaning attached to neither name, they are just labels - you might have defined the 'left' host to be the 'right' host vice versa.

Normally, you would use the exact same configuration file on both sides. Interpretation of that file is done by checking the local configuration. For example if it was stated in the configuration file that the IP address for 'left' is 1.2.3.4, the software assumes that it runs on the left node if it finds that IP address configured on one of its network devices. The same file is interpreted differently on the other node, as that hosts configuration differs.

The left, leftsubnet and leftnexthop (and the right... counterparts) determine the layout of the connection:

The leftid and leftrsasigkey are used in authenticating the left participant. The leftrsasigkey is the public key of the left participant (in the example above the RSA keys are shortened for easy display). The private key is stored in the /etc/ipsec.secrets file and should be kept secure.

The keys can be generated on both client and server with the command:

	# ipsec rsasigkey --verbose 2048 > rsa.key
					

The IKE-daemon of IPSEC is called pluto. It will authenticate and negotiate the secure tunnels. Once the connections is set up, the kernel implementation of IPSEC routes traffic through the tunnel if need be.

plutoload=%search and plutostart=%search tell the pluto daemon to search the configuration file for auto= statements. All connections with auto=add will be loaded in the pluto database. Connections with auto=start will also be started.

Troubleshooting

Network troubleshooting is a very broad subject with thousands of tools available. There are some very good books available on this topic, so we will limit our discussion to an overview of some of the tools which can be used to solve or detect network problems.

ifconfig

ifconfig checks the network interface configuration. Use this command to verify the user's configuration if the user's system has been recently configured or if the user's system cannot reach the remote host while other systems on the same network can.

When ifconfig is entered with an interface name and no other arguments, it displays the current values assigned to that interface. For example, checking interface eth0 system gives this report:

	$ ifconfig eth0
	eth0      Link encap:Ethernet  HWaddr 00:10:60:58:05:36  
	          inet addr:192.168.2.3  Bcast:192.168.2.255  Mask:255.255.255.0
	          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
	          RX packets:1398185 errors:0 dropped:0 overruns:0 frame:0
	          TX packets:1411351 errors:0 dropped:0 overruns:0 carrier:0
	          collisions:829 txqueuelen:100 
	          RX bytes:903174205 (861.3 Mb)  TX bytes:201042106 (191.7 Mb)
	          Interrupt:11 Base address:0xa000 
				

The ifconfig command displays a few lines of output. The third line of the display shows the characteristics of the interface. Check for these characteristics:

UP

The interface is enabled for use. If the interface is down, bring the interface up with the ifconfig command (e.g. ifconfig eth0 up).

RUNNING

This interface is operational. If the interface is not running, the driver for this interface may not be properly installed.

The second line of ifconfig output shows the IP address, the subnet mask and the broadcast address. Check these three fields to make sure the network interface is properly configured.

Two common interface configuration problems are misconfigured subnet masks and incorrect IP addresses. A bad subnet mask is indicated when the host can reach other hosts on its local subnet and remote hosts on distant network, but cannot reach hosts on other local subnets. ifconfig quickly reveals if a bad subnet mask is set.

An incorrectly set IP address can be a subtle problem. If the network part of the address is incorrect, every ping will fail with the no answer error. In this case, using ifconfig will reveal the incorrect address. However, if the host part of the address is wrong, the problem can be more difficult to detect. A small system, such as a PC that only connects out to other systems and never accepts incoming connections, can run for a long time with the wrong address without its user noticing the problem. Additionally, the system that suffers the ill effects may not be the one that is misconfigured. It is possible for someone to accidentally use your IP address on his own system and for his mistake to cause your system intermittent communications problems. This type of configuration error cannot be discovered by ifconfig, because the error is on a remote host. The arp command is used for this type of problem.

The ifconfig command can be used to set up multihomed network device. There are two ways a host can be multihomed.

Two Or More Interfaces To The Same Network: Devices such as servers or high-powered workstations may be equipped with two physical interfaces to the same network for performance and/or reliability reasons. They will have two IP addresses on the same network with the same network ID.

Interfaces To Two Or More Different Networks: Devices may have multiple interfaces to different networks. The IP addresses will typically have different network IDs in them.

ping

The basic format of the ping command on a Linux system is:

	$ ping [-c count] host
				

host

The hostname or IP address of the remote host being tested.

count

The number of packets to be sent in the test. Use the count field and set the value low. Otherwise, the ping command will continue to send test packets until you interrupt it, usually by pressing CTRL-C (^C).

To check that www.snow.nl can be reached from your workstation, send four packets with the following command.

	$ ping -c 4 www.snow.nl
	PING home.NL.net (193.67.79.250): 56 data bytes
	64 bytes from 193.67.79.250: icmp_seq=0 ttl=245 time=32.1 ms
	64 bytes from 193.67.79.250: icmp_seq=1 ttl=245 time=32.1 ms
	64 bytes from 193.67.79.250: icmp_seq=2 ttl=245 time=37.6 ms
	64 bytes from 193.67.79.250: icmp_seq=3 ttl=245 time=34.1 ms

	--- home.NL.net ping statistics ---
	4 packets transmitted, 4 packets received, 0% packet loss
	round-trip min/avg/max = 32.1/33.9/37.6 ms
				

This test shows a good wide-area network link to www.snow.nl with no packet loss and fast response. A small packet loss, and a round-trip time an order of magnitude higher, would not be abnormal for a connection made across a wide-area network. The statistics displayed by the ping command can indicate a low-level network problem. The key statistics are:

  • The sequence in which the packets are arriving, as shown by the ICMP sequence number (icmp_seq) displayed for each packet;

  • How long it takes a packet to make the round trip, displayed in milliseconds after the string time=;

  • The percentage of packets lost, displayed in a summary line at the end of the ping output.

If the packet loss is high, the response time is very high or packets are arriving out of order, there could be a network hardware problem. If you see these conditions when communicating over great distances on a wide area network, there is nothing to worry about. TCP/IP was designed to deal with unreliable networks, and some wide-area networks suffer from a high level of packet loss. But if these problems are seen on a local-area network, they indicate trouble.

On a local-network cable segment, the round-trip time should be near 0, there should be little or no packet loss and the packets should arrive in order. If these things are not true, there is a problem with the network hardware. On an Ethernet, the problem could be improper cable termination, a bad cable segment or a bad piece of active hardware, such as a hub, switch or transceiver.

The results of a simple ping test, even if the ping is successful, can help direct you to further testing toward the most likely causes of the problem. But other diagnostic tools are needed to examine the problem more closely and find the underlying cause.

route

To check the routing of a linux box, the route command is usually entered with no parameters or with -n to turn off ip-address to name resolution. For example route -n might show:

	$ route -n
	Kernel IP routing table
	Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
	192.168.11.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
	145.66.8.0      0.0.0.0         255.255.252.0   U     0      0        0 eth1
	0.0.0.0         145.66.11.254   0.0.0.0         UG    0      0        0 eth1
				

This host has two interfaces, one on subnet 192.168.11.0/24 the other on subnet 145.66.8.0/22. There is also a default route out on eth1 to 145.66.11.254 (denoted by the G under Flags and a Destination and Genmask of 0.0.0.0).

To be able to troubleshoot this information you need to know what the routing should be, perhaps by saving the routing information when the system is known to work.

The two most common mistakes are:

  • No network entry for an interface. When a network interface is configured a routing entry should be added that informs the kernel about the network that can be reached through the interface.

  • No default route (or two default routes). There should be exactly one default route. Note that two default gateways can go undetected for a long time because the routing could accidentally use the proper gateway.

In general, if there is a routing problem, it is better to first locate the part of the network where the problem originates, e.g. with ping or traceroute and then use route as part of the diagnostics.

traceroute

traceroute is a tool used to discover the links along a path. Path discovery is an essential step in diagnosing routing problems.

The traceroute is based on a clever use of the Time-To-Live (TTL) field in the IP packet's header. The TTL field is used to limit the life of a packet. When a router fails or is misconfigured, a routing loop or circular path may result. The TTL field prevents packets from remaining on a network indefinitely should such a routing loop occur. A packet's TTL field is decremented each time the packet crosses a router on its way through a network. When its value reaches 0, the packet is discarded rather than forwarded. When discarded, an ICMP TIME_EXCEEDED message is sent back to the packet's source to inform the source that the packet was discarded. By manipulating the TTL field of the original packet, the program traceroute uses information from these ICMP messages to discover paths through a network.

traceroute sends a series of UDP packets with the destination address of the device you want a path to. By default, traceroute sends sets of three packets to discover each hop. traceroute sets the TTL field in the first three packets to a value of 1 so that they are discarded by the first router on the path. When the ICMP TIME_EXCEEDED messages are returned by that router, traceroute records the source IP address of these ICMP messages. This is the IP address of the first hop on the route to the destination.

Next, three packets are sent with their TTL field set to 2. These will be discarded by the second router on the path. The ICMP messages returned by this router reveal the IP address of the second router on the path. The program proceeds in this manner until a set of packets finally has a TTL value large enough so that the packets reach their destination. Most implementations of traceroute default to trying only 30 hops before halting.

An example traceroute on linux looks like this:

	$ traceroute vhost2.cistron.net
	traceroute to vhost2.cistron.net (195.64.68.36), 30 hops max, 38 byte packets
	 1  gateway.kabel.netvisit.nl (145.66.11.254)  56.013 ms  19.120 ms  12.642 ms
	 2  145.66.3.45 (145.66.3.45)  138.138 ms  28.482 ms  28.788 ms
	 3  asd-dc2-ias-ar10.nl.kpn.net (194.151.226.2)  102.338 ms  240.596 ms  253.462 ms
	 4  asd-dc2-ipc-dr03.nl.kpn.net (195.190.227.242)  95.325 ms  76.738 ms  97.651 ms
	 5  asd-dc2-ipc-cr01.nl.kpn.net (195.190.232.206)  61.378 ms  60.673 ms  75.867 ms
	 6  asd-sara-ipc-dr01.nl.kpn.net (195.190.233.74)  111.493 ms  96.631 ms  77.398 ms
	 7  asd-sara-ias-pr10.nl.kpn.net (195.190.227.6)  78.721 ms  95.029 ms  82.613 ms
	 8  ams-icr-02.carrier1.net (212.4.192.60)  90.179 ms  80.634 ms  112.130 ms
	 9  212.4.192.21 (212.4.192.21)  49.521 ms  80.354 ms  63.503 ms
	10  212.4.194.42 (212.4.194.42)  94.528 ms  60.698 ms  103.550 ms
	11  vhost2.cistron.net (195.64.68.36)  102.057 ms  62.515 ms  66.637 ms
				

Again, knowing what the route through your network should be, helps to localize the problem. Note that not all network problems can be detected with a traceroute, because of some complicating factors. First, the router at some hop may not return ICMP TIME_EXCEEDED messages. Second, some older routers may incorrectly forward packets even though the TTL is 0. A third possibility is that ICMP messages may be given low priority and may not be returned in a timely manner. Finally, beyond some point of the path, ICMP packets may be blocked.

The results of a traceroute is a great tool to narrow down the possible causes of a network problem.

arp and arpwatch

arp is used to manipulate the kernel's ARP cache. The primary options are clearing an address mapping entry and manually setting one up. For debugging purposes, the arp program also allows a complete dump of the ARP cache.

If you know what the ARP address of a specific host should be, the dump may be used to determine a duplicate IP-address, but running arpwatch on a central system might prove more effective.

arpwatch keeps track of ethernet/IP address pairings. Changes are reported via syslog and e-mail.

arpwatch will report a changed ethernet address when a known IP address shows up with a new ethernet address. When the old ethernet address suddenly shows up again, a flip flop is reported.

Detection of duplicate IP addresses is very hard even with arpwatch. if, for example, a rogue host uses the IP address of the host running the arpwatch program or never communicates with it, a duplicate IP address will go unnoticed. Still, arpwatch is a very useful tool in detecting networking problems.

tcpdump

tcpdump is a program that enables network administrators to inspect in real-time every packet going through the network to which his or her computer is attached. This tool is typically used to monitor active connections or troubleshoot occasional network connectivity problems.

tcpdump can generate a lot of output, so it may be useful to limit the reported packets by specifying a port (e.g. tcpdump port 80). There are lots of other ways to specify which packets and what information we are interested in - see the manpage of tcpdump for more information.

nmap

nmap is a versatile tool for network exploration and security auditing. It's main use is as a portscanner, which also can identify running services, versions of the running services and OS types.

An example of this command and output is:

	$ nmap -A localhost

	Starting Nmap 6.25 ( http://nmap.org ) at 2013-07-04 04:01 CDT
	Nmap scan report for localhost (127.0.0.1)
	Host is up (0.00092s latency).
	Other addresses for localhost (not scanned): 127.0.0.1
	Not shown: 993 closed ports
	PORT     STATE SERVICE   VERSION
	22/tcp   open  ssh       OpenSSH 6.2p2 Debian 4 (protocol 2.0)
	| ssh-hostkey: 1024 f6:bf:2d:57:41:1b:fe:fa:d2:10:e0:2d:c3:89:c1:80 (DSA)
	| 2048 0d:11:fc:60:83:69:19:76:d1:fd:01:e5:7f:36:19:00 (RSA)
	|_256 0a:7b:0c:3a:86:c1:f7:63:6e:fb:f0:3c:62:42:c1:58 (ECDSA)
	25/tcp   open  smtp      Exim smtpd 4.80
	| smtp-commands: linux.mailserver Hello localhost [127.0.0.1], SIZE 52428800, 8BITMIME, PIPELINING, HELP, 
	|_ Commands supported: AUTH HELO EHLO MAIL RCPT DATA NOOP QUIT RSET HELP 
	53/tcp   open  domain
	| dns-nsid: 
	|_  bind.version: 9.8.4-rpz2+rl005.12-P1
	80/tcp   open  http      Apache httpd 2.2.22 ((Debian))
	|_http-title: Site doesn't have a title (text/html).
	111/tcp  open  rpcbind   2-4 (RPC #100000)
	| rpcinfo: 
	|   program version   port/proto  service
	|   100000  2,3,4        111/tcp  rpcbind
	|   100000  2,3,4        111/udp  rpcbind
	|   100024  1          40817/udp  status
	|_  100024  1          49956/tcp  status
	389/tcp  open  ldap      OpenLDAP 2.2.X - 2.3.X
	3000/tcp open  ntop-http Ntop web interface 4.99.3
	Service Info: Host: linux.mailserver; OS: Linux; CPE: cpe:/o:linux:linux_kernel

	Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
	Nmap done: 1 IP address (1 host up) scanned in 12.46 seconds
			

Note

Some of the nmap command options require root privileges, consult the NMAP(1) manpage for more information

wireshark

wireshark (known as Ethereal until a trademark dispute in the summer of 2006) is an open source network protocol analyzer. It allows you to examine data from a live network or from a capture file on disk. You can interactively browse the capture data, delving down into just that level of packet detail you need.

Wireshark has several powerful features, including a rich display filter language and the ability to view the reconstructed stream of a TCP session. It also supports hundreds of protocols and media types. A tcpdump-like console version named tethereal is included. One word of caution is that Ethereal has suffered from dozens of remotely exploitable security holes, so stay up-to-date and be wary of running it on untrusted or hostile networks (such as security conferences).

/usr/bin/lsof

The lsof command lists all open files on a system. Since Linux treats everything as a file, it also shows open network sockets. It can be used to find open ports on a system as well as determining the origin of a network connection.

Options of the lsof used for network troubleshooting.

-i

List IP sockets. The -i option also takes arguments to restrict the sockets listed, like -i tcp or -i 192.168.1.2.

-n

Do not resolve hostnames; can make lsof run faster.

-P

Do not resolve port names; can make lsof run faster.

+P

Resolve port name to protocol; default behaviour.

/bin/netstat

Like lsof,netstat is a program to list open ports on a system. It looks for the standard ports, but it also finds custom ports opened by applications, for instance netcat.

Frequently used options are:

-a

List all sockets.

-e

Extended mode.

-inet

List only IP connections

-l

Only show listening sockets

-n

Show IP numbers instead of resolving them to hostnames.

-p

Show the PID number and name of the process that is holding the socket.

-r

Show the kernel routing table. Is the same as the route command.

/usr/bin/nc

nc Is used for arbitrary TCP and UDP connections and listens. The command allows the user to set up, analyze and read connections between systems. It is a very useful tool in the troubleshooting toolbox. nc can be used to open up any port the user wants to be opened.

Because nc is capable of connecting and listening to any port it can be used to transfer files between systems that lack Samba, FTP or SSH etc.

if nmap is not available the nc command can be used to check for open ports: Run nc command with -z flag. You need to specify host name / ip along with the port range to limit and speedup operation. eg.

	# nc -z localhost 1-1023
			

/sbin/ip

The ip command is already discussed in the previous chapter.

Copyright Snow B.V. The Netherlands