CCIE Security: IPSec VPN Overview (IKEv1)

In this post, I'm going to go over a high level explanation of VPNs and specifically IPSec. This is going to be the first in a series of VPN posts focusing on the various types of VPNs one might see on the CCIE Security lab or on the job. I think it's important to have this overview because as you configure IPSec VPN or troubleshoot it, it'll help you to know what's going on under the covers of that configuration.

Note: I plan on doing about 10 or so blog posts on the various types of VPN so this is just the first. Welcome the deep deep rabbit hole that's been my life for the last 2 months. Hopefully I can explain some of these concepts to you in a way that makes it easier for you to grasp.

What are VPNs?

Let's talk about Virtual Private Networks (VPNs) for a moment before we jump into IPSec VPNs. One thing to get out of the way is that saying something is a VPN doesn't necessarily mean that it's encrypting traffic or "private." It just means there is some sort of separation. There are many many different types layer 2 VPNs out there such as MPLS, VPLS, VLANs, etc as well as many types of layer 3 VPNS such as VRFs, SSL, GRE, PPTP, etc. We could probably go down a deep rabbit hole on any of these types of VPNs but I merely wanted to illustrate that while we can say something is a VPN, it doesn't mean that it's providing any additional security except some basic separation.

Why IPSec?

While some of the above protocols can span between sites and provide some segmentation, they don't necessarily protect the data itself. The reasons for IPSec is to add that extra security. Some of the benefits of IPSec include:

Confidentiality - Encryption keeps your data private
Integrity - Built-in mechanisms to detect and discard any data that may have been altered in transit
Authenticity - Verification of who or what if sending the data

IPSec functions below the transport layer so it's completely transparent to applications and the end users. IPSec can be implemented in firewalls and routers. I personally have a preference for routers and as I dig into VPNs further in later blog posts, you'll see that there are some limitations with firewall VPN implementations that do not exist in routers.

Authentication Header (AH) vs Encapsulating Security Payload (ESP)

IPSec has two options that you can use: the lesser-used Authentication Header (AH) and the more popular Encapsulating Security Payload. Let me dig into the differences really quickly:

Authentication Header (AH)

Gives you anti-replay protection, data integrity and authenticates the data's origin - not confidentiality
Doesn't work with NAT
IP Protocol 51

Encapsulating Security Payload (ESP)

Gives you anti-replay protection, data integrity, authenticates the data's origin, and provides encryption
Uses something called NAT-T (NAT Traversal) to work with NAT
IP Protocol 50
Contains six parts:
- Security Parameter Index (SPI) - Tells the receiving device the group of security protocols the sender is using. This includes the algorithms, keys and the validity time for the keys
- Sequence Number - Increments by 1 each packet sent using the same SPI. Protects against replay attack.
- Payload data - Data carried in the packet
- Padding - This is used for certain types of encryption algorithms to maintain a certain amount of bytes
- Pad Length - How much of the payload is padding
- Next Header - Type of data carried and

As you can see from the bolded differences above, AH really isn't used for good reason. You get the same benefits from ESP without the limitations and with encryption. I am not going to dig deeper into AH vs ESP than that but wanted to explain that to you in the event any reader was wondering why I wasn't going to go over the configuration of AH in my blog posts.

Let's dig into the mechanics of how IPSec is set up. This is important to know as you are troubleshooting the various types of IPSec VPNs in the future as a lot, a lot of the troubleshooting will begin with Phase 1 and Phase 2 so it's important that you understand what they are.

Internet Key Exchange (IKE)

IKE provides a way to manage the key exchange, authenticate the peers and agree on a policy securely. IKE uses a protocol called ISAKMP to negotiate IPSec parameters between two peers. ISAKMP communicates on UDP port 500. This transport is fixed for UDP/500 on both the source and destination port of the packet. During the initial setup, the two VPN peers set up a bidirectional tunnel called the ISAKMP Security Association (SA) communication. After that, two unidirectional tunnels called the IPSec Security Associations (SA) are set up for communication the data. It's important to remember that the ISAKMP SA is a single bidirectional secure communcation channel but the actual encrypted LAN-to-LAN data is not sent over it. It's merely in place for the setup of the IPSec SAs in which it will encrypt and send the data. Let's break it down further.

Phase 1

This is where ISAKMP sets up a secure communication channel to be able to negotiate the next phase in a secure manner. Internet Security Association Key Management Protocol (ISAKMP) is to used negotiate IPSec parameters between the two peers. This phase can be done in one of two modes:

Main Mode - Requires 6 messages. Odd number messages always come from the initiator while even are from the responder. Considered more secure than Aggressive Mode. As with every communication, there's always someone that initates the traffic. We call this the initiator. The device that responds is referred to as the responder. I'm going to break down each packet so you understand what eart part of the communication does.
- MM #1 - This is the first message that the initiator sends to a responder. It includes something called the SA Proposal which should include the encryption algorithm, the type of authentication method, Diffie-Hellman group, and hashing algorithm. This might be one single proposal or multiple ones. The important part is that there is a matching one on the responder that they can decide on. The lifetime of the SA is also included in this message. If you don't explicitly configure it, the default is going to be 24 hours.
  
  Note: When configure ISAKMP policies, you can configure multiple policies with a number prioritize. When both peers are deciding on a policy, they are looking at it from top-to-bottom. Much like when an access list is evaluated, the peers will pick the first match and go with that. As a best practice, you should give your secure policies the lowest number to ensure they are chosen before a less secure policy.
- MM #2 - This message is sent from the responder to the initiator with the SA proposal that it chose.
- MM #3 - In this message, the initiator starts the Diffie-Hellman exchange. This is based on the Diffie-Hellman group sent in the proposal. This is the beginning of the key exchange.
- MM #4 - This will be the response to MM #3. The responder will send it's own key to the initiator. At this point, encryption should be established for the ISAKMP SA channel
- MM #5 - At this point, there should be encryption and encryption keys have been shared. The initiator will send this message to authenticate the session. This message will often contain the IP address of the router and sometimes the distinguished name or hostname.
- MM #6 - The responder will send back a similar packet and authenticate the session. At this point, the ISAKMP SA channel has been established.
Aggressive Mode - Only requires 3 messages which will give you a faster connection but inherently less secure because the responder no longer has to authenticate itself first in any exchange and one can potentially brute force the pre-shared key. I'll break Aggressive Mode down further but it's unlikely you'll see this as often.
- AM #1 - In this message, the initiator will send all the information that was contained in MM #1-3 and some information from MM #5 so essentially the policies, key generation information and identity information.
- AM #2 - This will send all the same information contained in MM # 2, 4, and 6 which includes the key generation information, identity information and authentication information.
- AM #3 - This contains the authentication and identity information that is contained in MM #5

Phase 2

In this phase, the negotiation is protected between the two peers thanks to the ISAKMP SA that's already been established and the end goal of this phase is to have two unidirectional channels between the peers set up to pass traffic in a secure manner over an insecure network. This phase uses something called Quick Mode to establish these IPSec SAs. Let's dig into the messages that are exchanged:

QM #1 - The peer will send an IPSec Proposal this time which will include agreed upon algorithms for encryption, integrity, and what traffic is to be secured or encrypted (Proxy ID). The traffic that is to be secured will typically be defined as part of an ACL. Configuring the IPSec proposal is completely different than configuring the ISAKMP policy. the algorithms you use here can be completely different than phase 1.

Note: I'm pointing out the ACL here and approaching this from the very simple view of L2L VPN with crypto maps. In later posts, I'll go deeper into tunnels and other types of VPN which don't require ACLs.
QM #2 - This is the reply from the responder to the initiator which will include the chosen proposal and the Proxy IDs. The Proxy ID will probably be the inverse of the first Proxy ID that was sent.
QM #3 - This is a final acknowledgement. After this, there should be two unidirectional secure channels from one peer to the other peer.

One thing to note is that there is a timeout for the IPSec SA where the encryption key is changed. The default for this is 1 hour. Another thing you can enable with IPSec is Perfect Forward Secrecy (PFS) which will help frequent changes of the encryption keys and if enabled, it can help cycle the keys out more regularly but when this is enabled, you might see more issues between different vendors. If the PFS group is configured in the IPSec configuration, it should match on both side and it should not be the same as the Diffie-Hellman group.

Encryption Modes

There are two IPSec modes to consider and some VPN types will only support one of these modes.

Tunnel Mode

This mode is considered more secure than Transport Mode because it will encrypt both the payload and the original header. A new IP header will be placed on the packet since ESP will be encapsulating the whole thing. The downside of this is that since a new IP header will need to be created, this will add to the overhead of the original packet. This mode is supported by something call NAT-T (NAT Traversal) and is the default mode for Cisco routers.

Transport mode

In this mode, it encrypts only the payload of each packet while the original packet header is left along. ESP will still be the transport but the entire original packet will not be encrypted. This is typically not seen as secure on it's own because the original IP header is in cleartext and some information can be gleamed from that but there are practical applications for this which we will go into in later posts. This mode is often used with tunnel interfaces which we will cover in later posts. A good thing about Transport Mode is that it eliminates the need for another IP header which takes away from the added overhead on the packet but a downside is that it doesn't natively support NAT-T since it will fail checksum verifications.

NAT Traversal (NAT-T)

As I stated a few paragraphs back, I like routers a lot more for VPN for a number of reasons. There's just a lot more options and flexibility but in configuring VPN from the router, you might find yourself or your peer behind a NAT or PAT device. ESP encrypts everything inside of the packet including the inner TCP/UDP data. While ESP is an IP protocol, it doesn't have any port information in the same way that ISAKMP uses UDP port 500 for it's transport. PAT won't be able to use a unique port as it's trying to process ESP traffic or add it to it's binding database. This will cause IPSec to never form if it's left on it's own but thankfully, IPSec has a mechanism to help called NAT-T. This mechanism detects if there are any NAT devices in the path of the peers and if both peers support NAT-T. NAT-T discovery is done in MM#3 and #4 where both peers send a hash in those messages and if the hashes they send do not match, that means that a NAT device is in the path.

When this happens, NAT-T will change the ISAKMP transport at MM#5 and #6 from UDP port 500 to UDP port 4500 before we get to the IPSec SA creation. Phase 2 will also complete inside UDP port 4500. This provides a port that the PAT database can bind to the session and IPSec will be able to form the IPSec SAs.

After NAT-T - A happy packet that can now be PAT-ed

With that, I'm going to wrap up this blog post. Expect a lot more VPN posts from me in the future.