VPN/IPsec & SSH PQC
Explore post-quantum key exchange in IKEv2, SSH, and WireGuard protocols.
IKEv2 Handshake Overview
(Internet Key Exchange version 2, RFC 7296) is the key negotiation protocol for VPNs. It establishes Security Associations (SAs) through a two-phase handshake: IKE_SA_INIT for key exchange and IKE_AUTH for identity verification.
The KE payload carries public values. Both parties derive the same shared secret (SKEYSEED) from their DH exchange, nonces, and SPIs, then derive encryption and integrity keys for the IKE SA.
in IKEv2
The IETF draft draft-ietf-ipsecme-ikev2-mlkem defines how to integrate ML-KEM () into IKEv2 using the Additional Key Exchange (AKE) framework (RFC 9370). This enables hybrid key exchange without modifying the core IKEv2 state machine.
“The initiator sends an ML-KEM encapsulation key in an Additional Key Exchange payload during IKE_INTERMEDIATE. The responder encapsulates against this key and returns the ciphertext. The resulting shared secret is combined with the classical DH secret using the IKEv2 key hierarchy.”
— draft-ietf-ipsecme-ikev2-mlkem-01
ECP-256 or MODP-2048 in KE payload during IKE_SA_INIT. Quantum-vulnerable.
ML-KEM-768 encapsulation key (1,184 B) in IKE_INTERMEDIATE. Adds one round trip.
SKEYSEED derived from both DH and KEM shared secrets via PRF. Secure if either holds.
Key Exchange with
OpenSSH has been a pioneer in deploying post-quantum key exchange. Since version 8.5 (March 2021), it has shipped with sntrup761x25519-sha512 — a hybrid of Streamlined NTRU Prime and X25519. In OpenSSH 9.9 (September 2024), NIST-standard mlkem768x25519-sha256 was added.
sntrup761x25519-sha512sntrup761x25519-sha512mlkem768x25519-sha256SSH hybrid KEX works by concatenating the classical and PQC public keys in SSH_MSG_KEX_ECDH_INIT, and concatenating the classical share and KEM ciphertext in SSH_MSG_KEX_ECDH_REPLY. The shared secret is derived from both components.
& PQC
WireGuard uses the Noise IK protocol framework with a fixed cipher suite: , ChaCha20-Poly1305, BLAKE2s. This minimal design makes WireGuard fast and auditable, but also means it has zero — algorithms cannot be negotiated.
WireGuard's fixed X25519 is quantum-vulnerable. Adding PQC requires modifying the protocol or layering a separate negotiation on top.
runs a separate PQC key exchange (ML-KEM + Classic McEliece) and feeds the resulting PSK into WireGuard's pre-shared key slot.
The Rosenpass approach preserves WireGuard's simplicity while adding quantum resistance. The PSK is rotated periodically, and WireGuard's security holds even if Rosenpass is compromised (defense in depth).
Protocol Size Impact
Post-quantum key exchange significantly increases handshake sizes across all protocols. ML-KEM-768 adds approximately 1,184 bytes for the encapsulation key and 1,088 bytes for the ciphertext. For protocols like WireGuard that use UDP, this can cause IP fragmentation issues.
| Protocol | Classical | Hybrid | Increase |
|---|---|---|---|
| IKEv2 | 1,400 B | 3,768 B | +169% |
| SSH | 984 B | 3,296 B | +235% |
| WireGuard | 304 B | 6,800 B | +2137% |
| TLS 1.3 | 1,200 B | 3,500 B | +192% |
WireGuard sees the largest relative increase (22x) because its classical handshake is extremely compact. IKEv2 handles fragmentation explicitly (RFC 7383) over UDP, while SSH handles larger payloads natively because it relies on TCP transport.
Authentication: The Missing Half
While PQC Key Exchange (ML-KEM) secures connections against future decryption, Authentication using secures against active attackers. Both IKEv2 and SSH uniquely require multiple layers of authentication that must be migrated.
In addition to the AKE payloads, draft-ietf-ipsecme-ikev2-mldsa updates the IKE_AUTH phase to support (). The Auth payload verifies identities using post-quantum certificates.
SSH requires PQC host keys (server identity) and user keys (UserAuth). While OpenSSH 9.9 supports ML-KEM, standardizing ML-DSA keys for host and user authentication is still an active effort.
Control Plane vs. Data Plane
The migration to PQC only applies to the Control Plane (key exchange & auth). The actual Data Plane (the VPN tunnel) already uses symmetric algorithms like AES-GCM or ChaCha20-Poly1305. These algorithms are naturally quantum-resistant, requiring only standard key lengths (256-bit) to be secure against Grover's algorithm.
Related Resources
Step through IKEv2 and SSH handshakes, compare classical vs hybrid vs pure PQC modes.