Quick Summary
- A VPN creates an encrypted tunnel between two endpoints, hiding your data from eavesdroppers
- IPsec uses IKE for key exchange and ESP for encrypting packet payloads
- Tunnel mode encapsulates the entire original packet inside a new encrypted packet with new headers
What is a VPN?
A Virtual Private Network (VPN) creates a secure, encrypted connection over an untrusted network -- typically the public internet. Imagine you need to send a confidential letter through a public postal system. Without protection, any postal worker along the route could read it. A VPN is like sealing your letter inside an opaque, tamper-proof envelope that only the intended recipient can open. The postal workers can see the envelope, but not the letter inside.
VPNs serve two primary purposes: privacy (preventing eavesdroppers from reading your traffic) and access (connecting to resources as if you were on a private network). When you connect to your company's VPN from home, your laptop creates an encrypted tunnel to the corporate network. From the perspective of internal servers, your traffic appears to originate from inside the office -- even though you are miles away.
There are several VPN protocols, but IPsec (Internet Protocol Security) is the most widely deployed for site-to-site VPNs and is built into every major operating system. Other popular protocols include WireGuard (modern and fast), OpenVPN (open-source and flexible), and L2TP (often combined with IPsec). Each makes different tradeoffs between speed, security, and ease of configuration.
Interactive: VPN On vs. VPN Off
Toggle the VPN switch to see how your packet looks with and without encryption. Notice how the data payload changes.
How IPsec Works
IKE Phase 1: Establishing Trust
Internet Key Exchange (IKE) is the protocol that sets up the secure tunnel before any data flows. In Phase 1, the two endpoints negotiate encryption algorithms (like AES-256), authentication methods (pre-shared keys or certificates), and Diffie-Hellman groups for key exchange. They authenticate each other's identity and establish a shared secret key using the Diffie-Hellman algorithm -- a mathematical process that lets two parties create a shared secret over an insecure channel without ever transmitting the secret itself.
IKE Phase 2: Building the Tunnel
Phase 2 uses the secure channel from Phase 1 to negotiate the parameters for the actual data tunnel (called a Security Association, or SA). This includes which traffic to protect (defined by source/destination subnets and ports), which encryption and hashing algorithms to use for the data, and the lifetime of the tunnel before keys must be renegotiated. Phase 2 produces the session keys that will encrypt your actual data packets using ESP.
ESP: Encrypting the Data
Encapsulating Security Payload (ESP) is the protocol that does the actual encryption and authentication of data packets. Each packet gets an ESP header containing a Security Parameter Index (SPI) that identifies which Security Association to use, a sequence number to prevent replay attacks, and the encrypted payload. ESP also appends an Integrity Check Value (ICV) -- a cryptographic hash that lets the receiver verify the packet was not tampered with in transit. ESP provides confidentiality, integrity, and anti-replay protection.
Tunnel vs. Transport Mode
IPsec operates in two modes. Tunnel mode encapsulates the entire original IP packet (headers and all) inside a new IP packet with new headers. This is used for site-to-site VPNs where the original source and destination addresses must be hidden. Transport mode only encrypts the payload of the original packet, leaving the original IP headers intact. This is used for host-to-host communication where the endpoints are the actual communicating devices (not gateways).
Types of VPN Deployments
Site-to-Site VPN
Connects two entire networks over the internet. The VPN gateways at each site handle encryption transparently -- devices on either network communicate as if they were on the same LAN. Used by enterprises to link branch offices to headquarters.
Remote Access VPN
Connects individual users to a corporate network. A VPN client on the user's device creates a tunnel to the corporate VPN gateway. All traffic (or selected traffic) is routed through the tunnel, giving the user access to internal resources.
Cloud VPN
Connects on-premises networks to cloud provider networks (AWS, Azure, GCP). Uses IPsec tunnels to extend the corporate network into the cloud, allowing hybrid architectures where some workloads run on-premises and others in the cloud.
IPsec vs. WireGuard vs. OpenVPN
IPsec is the industry standard, supported by every major network vendor and operating system. Its strength is interoperability -- a Cisco router can establish an IPsec tunnel with a Juniper router without issues. However, IPsec is complex to configure, with dozens of parameters to negotiate and a sprawling set of RFCs defining its behavior.
WireGuard is a modern VPN protocol that prioritizes simplicity and performance. Its entire codebase is roughly 4,000 lines of code (compared to hundreds of thousands for IPsec implementations). WireGuard uses state-of-the-art cryptography (ChaCha20, Curve25519, BLAKE2) and achieves significantly higher throughput than IPsec or OpenVPN in benchmarks. It is now built into the Linux kernel and is rapidly gaining adoption.
OpenVPN is an open-source VPN that operates over SSL/TLS. It runs in userspace (not the kernel), making it more portable but generally slower. Its main advantage is flexibility -- it can traverse firewalls and NAT devices easily because it can operate over TCP port 443, which looks like normal HTTPS traffic to firewalls.
Common VPN Pitfalls
DNS Leaks
Even with a VPN active, DNS queries may bypass the tunnel and go directly to your ISP's resolver, revealing which sites you visit. Always configure your VPN to route DNS through the tunnel or use a trusted resolver.
Split Tunneling Risks
Split tunneling routes only corporate-bound traffic through the VPN, sending everything else directly to the internet. This improves performance but means the user's device is simultaneously connected to the internet and the corporate network -- a potential bridge for attackers.
Weak Pre-Shared Keys
Using simple, short, or shared-across-sites pre-shared keys (PSKs) for IKE authentication is a common and dangerous shortcut. If the PSK is compromised, an attacker can decrypt all traffic. Certificate-based authentication is stronger.
Frequently asked questions about IPsec VPN
What does IPsec stand for?
IPsec stands for Internet Protocol Security. It is a suite of protocols that authenticates and encrypts IP packets, most commonly using IKE for key exchange and ESP to protect the payload, and it is built into every major operating system.
How does an IPsec VPN work?
Two endpoints first run IKE to authenticate each other and negotiate keys, then traffic between them is wrapped in ESP packets. In tunnel mode the entire original IP packet is encrypted and placed inside a new outer IP header, so intermediate networks see only an opaque, encrypted blob.
What is the difference between IPsec tunnel mode and transport mode?
Tunnel mode encrypts the entire original packet including its headers and encapsulates it in a new outer packet — this is what site-to-site VPNs use. Transport mode encrypts only the payload and leaves the original IP header visible, mostly used between two hosts rather than gateways.
Which is better: IPsec, WireGuard, or OpenVPN?
IPsec is the most widely interoperable and the standard choice for site-to-site VPNs between vendor appliances. WireGuard is much simpler, faster, and easier to configure but newer. OpenVPN runs over TLS and is flexible across firewalls, but slower than the other two.
Why does my IPsec VPN keep dropping?
The usual culprits are MTU and PMTU issues (the extra ESP overhead causes fragmentation), NAT in front of one endpoint without NAT-Traversal enabled, mismatched IKE or ESP proposals between sides, and lifetime expiry where rekeying fails. Check both endpoints' logs for the exact phase where negotiation stalls.
Try the MPLS VPN lab
You just learned how IKE, ESP, and tunnel-mode encapsulation protect traffic between sites. IPsec is not modelled in the app, but you can connect two customer sites across an MPLS backbone and trace the labels that carry traffic between them: a provider VPN that separates customers rather than encrypting their traffic.
Launch the MPLS VPN lab →