Introduction to IPsec

What is IPsec?

IPsec is a set of extensions to the IP protocol, described in RFC-2401◳, designed to provide interoperable, high quality, cryptographic security at the IP layer. The IPsec RFC describes two distinct protocols, both of which can be used in two distinct modes. The first mode provides security and authenticity assurance services to the data transmitted over a network. This mode of operation is usually referred to as transport mode. The second mode of operation is used to provide secure tunnels, often over the Internet, to connect either a single computer to a remote network or to join two remote networks together to form a single virtual private network (VPN). This mode of operation is usually referred to as tunnel mode.

IPsec protocols

Authentication Header (AH)

The IP Authentication Header (AH), described in RFC-2402◳, is used to provide connectionless integrity and data origin authentication for IP datagrams, and to provide protection against replays. Adding an Authentication Header to an IP datagram does not ensure the security of the payload of the packet against snooping by external entities. To encrypt the payload of an IP datagram the Encapsulating Security Payload (ESP) protocol described below should be used.

Encapsulating Security Payload (ESP)

The IP Encapsulating Security Payload (ESP), described in RFC-2406◳, is used to provide confidentiality, data origin authentication, connectionless integrity, an anti-replay service (a form of partial sequence integrity), and limited traffic flow confidentiality. Unlike the AH protocol, described briefly above, the payload of the IP datagram is encrypted when using the ESP protocol which makes it particularly well suited to securing plain-text protocols which would otherwise be vulnerable to snooping by external entities.

IPsec modes

Transport mode

IPsec in Transport Mode can provide a secure authenticated connection between two end-points by adding AH and ESP headers to the original IP packet. If the ESP protocol is used then the data portion of the IP packet is also be encrypted. When using IPsec in Transport Mode the source and destination addresses are secured making the resulting datagrams incompatible with any packet mangling technology such as Network Address Translation (NAT).

Tunnel mode

IPsec in Tunnel Mode encapsulates the original IP packet, which may be secured by adding AH and ESP headers as above, adding a new IP header to create a Virtual Private Network (VPN). As the new IP header is not secured by IPsec in this mode of operation the resulting datagrams are able to pass through packet mangling technology such as Network Address Translation (NAT) unaffected.

IPsec policy

Security Policy Database (SPD)

The Security Policy Database (SPD) specifies what services are to be offered to IP datagrams and in what fashion those services are to be implemented. The SPD is consulted during the processing of all traffic, both in-bound and out-bound, and specifies the action to be taken to secure that traffic. Actions can be either discard (in which case the packets associated with this entry are discarded), bypass (in which case the packets associated with this entry are allowed to pass without the IPsec protocol being used) or apply (in which case packets associated with this entry will have the IPsec transforms specified in this entry applied).

Security Association Database (SAD)

The Security Association Database (SAD) stores the parameters for each Security Association (SA). Security Associations can either be managed manually, using a utility such as the setkey application, or populated automatically by a daemon such as racoon, both of which are described below and used in this guide.

IPsec on Linux

Kernel

In the Linux implementation of IPsec the kernel maintains both the SPD and SAD as well as performing all the cryptographic, compression and IP protocol functions. As usual the Linux kernel tries to separate implementation and policy management leaving policy management functions to user-space applications which are required to connect to the IPsec implementation in the kernel using PF_KEY sockets.

setkey application

The setkey application provides an interface to the SPD and SAD maintained by the kernel and can be used to manually manage both the Security Policy and Security Association databases. Usually however the setkey application is only used to populate the Security Policy Database with another user-space application providing automated key exchange facilities used to manage the Security Association Database.

racoon daemon

The racoon daemon provides a method of automatically populating the SAD using kernel events triggered by matching SPD entries using the Internet Security Association and Key Management Protocol (ISAKMP) described in RFC-2408◳, and the OAKLEY Key Determination Protocol described in RFC-2412◳. In conjunction with the setkey application described above the racoon daemon can be used to automatically exchange keys using a variety of key sharing methods such as PSK (Pre Shared Key) or X.509 certificates to provide a completely automated IPsec solution.