IPsec VPN, explained

IPsec is the protocol suite behind most site-to-site and always-on VPN tunnels. Here is what it is, how IKE and ESP fit together, tunnel vs transport mode, and the ports it uses.

Tunnel mode wraps the entire original packet: ESP encrypts the inner IP header and payload, and the outer header routes the tunnel between gateways. Transport mode (host-to-host) keeps the original IP header instead.

An IPsec VPN secures traffic at the IP layer using the IPsec protocol suite: it authenticates the two ends, negotiates keys, and then encrypts and integrity-protects every packet that crosses the tunnel. It is the standard behind most site-to-site VPNs and a common choice for always-on remote access, because it runs below the application and protects everything above it transparently.

Unlike an SSL/TLS VPN, which secures a session for applications that speak TLS, IPsec protects IP packets themselves. That is why it is the natural fit for joining networks: routers and hosts behind an IPsec tunnel need no VPN awareness at all.

What IPsec is made of

IPsec is not one protocol but a suite. Three parts matter in practice: IKE (Internet Key Exchange) negotiates who the peers are and what keys they use; ESP (Encapsulating Security Payload) does the actual encryption and integrity protection of the data; and Security Associations (SAs) are the agreed parameters — algorithms, keys, lifetimes — for each direction of the tunnel.

There is also AH (Authentication Header), which provides integrity and authentication but no encryption. ESP is used far more often because it does both, so most deployments you will see are ESP-based. The current key-exchange standard is IKEv2, defined in RFC 7296, and ESP is defined in RFC 4303.

How the tunnel comes up

An IPsec tunnel is established in two negotiation phases. In IKE phase 1, the peers authenticate each other — with a pre-shared key or certificates — and build a secure channel between themselves. In phase 2, they use that channel to negotiate the IPsec SAs that will actually protect the data traffic.

Once the SAs exist, ESP encrypts each packet, adds its header and trailer, and the packet is routed to the far gateway, which decrypts it and forwards the original packet onward. Ephemeral Diffie-Hellman keys give perfect forward secrecy, so capturing one session's key does not expose past or future traffic. Modern deployments use AES-GCM or ChaCha20-Poly1305 for encryption.

Tunnel mode vs transport mode

IPsec has two modes. Tunnel mode encrypts the entire original IP packet and wraps it in a new outer IP header, so the real source and destination are hidden inside the tunnel — this is what site-to-site and remote-access VPNs use. Transport mode encrypts only the payload and keeps the original IP header, which suits host-to-host protection on a network you already trust.

The diagram above shows tunnel mode: ESP encrypts the inner IP header, transport header and payload, while the outer header carries the packet between gateways. If you are building a VPN rather than protecting a single host-to-host link, tunnel mode is almost always the mode you want.

Ports, NAT and firewalls

IKE negotiation runs over UDP port 500. ESP itself is not a TCP or UDP protocol — it is IP protocol number 50 — which is why it struggles to cross NAT devices that expect port numbers. The fix is NAT traversal: when a NAT is detected, IPsec wraps ESP inside UDP on port 4500, as defined in RFC 3948. So in practice you open UDP 500 and UDP 4500.

This is also the honest answer to a lot of old router folklore about 'IPsec passthrough'. NAT-T solved ESP-over-NAT properly, which is why the passthrough toggle matters far less than it once did — see what VPN passthrough is for the detail.

Where IPsec fits — and where it does not

IPsec is the right tool for site-to-site tunnels, for always-on device connectivity, and anywhere you need to protect all IP traffic between two points. It is heavier to configure than a TLS-based remote-access VPN and can be fiddlier through restrictive networks. For the decision between them, read IPsec vs SSL VPN; for how IPsec sits in the bigger picture, the business VPN guide puts it next to the alternatives.

And remember what IPsec does not do: it secures the transit, not the endpoints or the data at rest. A correctly configured IPsec tunnel to a compromised host is still a tunnel to a compromised host.

Frequently asked questions

What is an IPsec VPN in simple terms?

It is a VPN that secures traffic at the IP layer: it authenticates the two ends, exchanges keys with IKE, and encrypts every packet with ESP. Because it works below applications, it protects all IP traffic between two points transparently, which is why it is the standard for site-to-site VPNs.

What ports does IPsec use?

IKE key negotiation uses UDP port 500. When a NAT device is in the path, IPsec encapsulates ESP inside UDP port 4500 (NAT traversal). ESP itself is IP protocol 50, not a TCP/UDP port, which is why NAT-T on UDP 4500 exists.

What is the difference between tunnel mode and transport mode?

Tunnel mode encrypts the whole original packet and adds a new outer IP header, hiding the real addresses inside the tunnel — used by site-to-site and remote-access VPNs. Transport mode encrypts only the payload and keeps the original header, used for host-to-host protection.

Is IPsec more secure than an SSL VPN?

Not inherently. Both can use strong modern ciphers. IPsec protects all IP traffic and suits site-to-site; SSL/TLS VPNs traverse firewalls and NAT more easily and suit clientless remote access. Security in practice depends far more on configuration and patching than on which suite you pick.

What is the difference between ESP and AH?

ESP (Encapsulating Security Payload) provides encryption plus integrity and authentication; AH (Authentication Header) provides integrity and authentication but no encryption. Almost all modern IPsec VPNs use ESP because it protects confidentiality as well.