FortiGate IPsec VPN — Site-to-Site and Dial-Up

The one-sentence mental model

Phase 1 builds a locked room for the negotiators to talk in. Phase 2 builds the encrypted pipes traffic actually flows through. Both are just Security Associations (SAs) — Phase 1 SA is for the control plane, Phase 2 SAs are for the data plane. If Phase 1 doesn’t come up, nothing works; if Phase 2 doesn’t come up, Phase 1 is fine but no traffic flows.

The negotiation flow

Phase 1 (IKE / IKEv2)
  Proposals: encryption + hash + DH group + lifetime
  Authentication: PSK  or  X.509 certificate
  Peer identity: IP address / FQDN / user-fqdn
    ↓ (success)
Phase 2 (IPsec)
  Proposals: encryption + hash + PFS DH group + lifetime
  Selectors: local subnet + remote subnet
    ↓ (success)
Tunnel is up. Traffic that matches the selectors gets encrypted.

Route-based vs policy-based

Route-based (FortiOS default):

• Creates a virtual interface (e.g., to-hq-tunnel) that behaves like a Layer 3 interface.
• You add a static route for the remote subnet pointing to the virtual interface.
• You write a firewall policy allowing traffic between local LAN and the virtual interface.
• Works with dynamic routing (BGP, OSPF), SD-WAN member selection, SLA-aware failover.

Policy-based (legacy):

• The firewall policy itself specifies “action: IPSEC” and picks the tunnel.
• No routing table involvement.
• No dynamic routing, no SD-WAN. Every real deployment today uses route-based.

Site-to-site config sketch (route-based)

Phase 1:

config vpn ipsec phase1-interface
  edit "hq-tunnel"
    set interface "wan1"
    set peertype any
    set net-device disable
    set proposal aes256-sha256
    set dhgrp 14
    set remote-gw 198.51.100.10
    set psksecret "SharedSecret123"
  next
end

Phase 2:

config vpn ipsec phase2-interface
  edit "hq-tunnel-p2"
    set phase1name "hq-tunnel"
    set proposal aes256-sha256
    set dhgrp 14
    set src-subnet 10.0.0.0/24
    set dst-subnet 10.1.0.0/24
  next
end

Route + firewall policy:

config router static
  edit 10
    set dst 10.1.0.0/24
    set device "hq-tunnel"
  next
end

Then a firewall policy: internal → hq-tunnel, source LAN, destination remote LAN, action ACCEPT.

Dial-up VPN

Used when the peer has a dynamic public IP (or is behind CGN NAT). FortiGate accepts inbound Phase 1 from any peer that presents a matching identity + PSK/cert:

• set remote-gw 0.0.0.0
• set type dynamic
• set peertype one|any|dialup
• Peer authenticates by FQDN or user-FQDN identity, not by IP.

Common use: remote branch FortiGates behind ISP CGN, road-warrior FortiClients.

NAT-Traversal

If either peer sits behind NAT, IPsec ESP breaks (NAT rewrites the source IP → ESP integrity check fails). NAT-T detects this and encapsulates ESP inside UDP 4500:

• IKE starts on UDP 500.
• Peers exchange NAT-D payloads. If NAT detected, both switch to UDP 4500.
• ESP is then wrapped in UDP 4500 for the data plane too.

Firewall you’re going through must permit UDP 500 + UDP 4500 outbound (and inbound if you’re the concentrator).

Diagnostics

The five commands that solve 90% of IPsec tickets:

diagnose vpn ike gateway list                # phase 1 state
diagnose vpn tunnel list                     # phase 2 SAs, byte counters
diagnose debug application ike -1
diagnose debug enable                        # live IKE negotiation trace
diagnose debug reset

Common messages:

• no matching gateway found for negotiation → wrong peer identity or proposal
• PSK auth failed → PSK mismatch
• no proposal chosen → Phase 1 or Phase 2 encryption/hash/DH mismatch
• NO-PROPOSAL-CHOSEN in the responder side → check selectors on Phase 2

Common exam / real-world mistakes

1. Mismatched proposals. Both sides must list at least one identical (encryption, hash, DH group). Simplest is to standardize on aes256-sha256 + DH 14.
2. Wrong Phase 2 selectors. If your local Phase 2 says 10.0.0.0/24 but the remote side says 10.0.0.0/16, they don’t match. Both sides’ selectors must be identical.
3. Forgetting the return-path firewall policy. You wrote LAN → tunnel; you also need tunnel → LAN if traffic starts from the remote side.
4. NAT-T disabled with NAT in path. Symptom: Phase 1 succeeds but Phase 2 traffic never flows. Enable NAT-T on both sides.
5. Using policy-based when you need SD-WAN. SD-WAN only picks members for route-based tunnels. If you’re on policy-based, switch.
6. Not accounting for PFS. Enabling Perfect Forward Secrecy on one side but not the other = no proposal chosen. Match both sides’ PFS DH group.

Cheat strip

Phases    P1 = IKE  (control channel)
          P2 = IPsec SA (data channel)

Modes     route-based (default)   → virtual interface + static route
          policy-based (legacy)   → action: IPSEC in firewall policy

Auth      PSK  |  X.509 cert
Peer ID   IP  |  FQDN  |  user-FQDN
Dial-up   remote-gw 0.0.0.0 + type dynamic

NAT-T     auto-detect. UDP 500 → 4500 if NAT in path.
Firewall  permit UDP 500 + UDP 4500.

Debug     diagnose vpn ike gateway list
          diagnose vpn tunnel list
          diagnose debug application ike -1  +  diagnose debug enable