I have a couple of IPSEC server-to-server (transport) policies configured
between Windows Server 2016 hosts and authenticated by Kerberos in our AD environment. They work as I would expect them to work, except for one thing: the ESP packets are not encapsulated in udp/4500 if none of the hosts is behind a NAT device.
The presence of NAT is presumably determined in IKE Phase 1 via the mechanism described inRFC 3947 section 3.2, the result is IP-ESP (Protocol 50) packets being sent out:
The NAT detection mechanism is apparently not fit to determine if the network in between is filtering ESP, so if it does, IPSEC ESP tunnels fail to establish although they would be establishing just fine over UDP-ESP encapsulation.
I have tried searching the internets, but most contributions on this topic deal with the L2TP VPN tunnel configuration where the server is behind a NAT device, which does not fit my scenario well. Nonetheless, I tried setting AssumeUDPEncapsulationContextOnSendRule in HKLM\CurrentControlSet\Services\PolicyAgent to
2 as suggested by most of the sources - but this does not seem to change anything, ESP packets still get sent out unencapsulated.
How could I force or favor UDP-ESP encapsulation for my server-to-server IPSEC connections?