Extended Ping and Traceroute on Cisco IOS

Mental model

A ping is a question and an answer. The router sends an ICMP Echo Request and waits for an Echo Reply (see ICMP for the message types). When you see !!!!!, two things worked: the request reached the target, and the target knew how to send the reply back to the request’s source address.

That second part is the one people forget. A failed ping only tells you the round trip broke somewhere. The request might never have arrived, or it arrived and the reply had no route home. Ping alone cannot tell you which.

On a router, a plain ping normally uses the outgoing interface toward the target as its source. So it tests the path from R1’s WAN interface, and says nothing about whether the far end can reach the LAN behind R1, which is where your users live. Extended ping lets you choose the source. Traceroute shows where along the path things stop.

Reading ping output

R1# ping 192.168.1.10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.10, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/2 ms

The defaults are right there: 5 echoes, 100 bytes, 2 second timeout. Each character is one echo. Cisco documents these:

Why the first one is often a dot. If R1 has no ARP entry for the next hop (or a directly connected target), it sends an ARP request and drops that first echo. By the second echo the MAC address is known and everything flows. So .!!!! on the first try is normal. Ping again: you should get !!!!!.

Extended ping

Type ping with no arguments in privileged EXEC and IOS walks you through every option:

R1# ping
Protocol [ip]:
Target IP address: 10.3.3.3
Repeat count [5]: 100
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: Loopback0
Type of service [0]:
Set DF bit in IP header? [no]: yes
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:

Brackets show the default; Enter accepts it. The fields that matter most:

• Repeat count: send 100 or 1000 to catch intermittent loss that 5 echoes would miss.
• Datagram size: the packet size in bytes.
• Source address or interface: an IP address on this router or an interface name. This is the big one.
• Set DF bit: tells routers along the path not to fragment the packet.
• Sweep range of sizes: IOS then asks for Sweep min size [36], Sweep max size [18024] and Sweep interval [1].

The common options also fit on one line:

R1# ping 10.3.3.3 source Loopback0 repeat 100 size 1500 df-bit

MTU testing. Send 1500 bytes with DF set. If a link along the path has a smaller MTU, the router in front of it cannot fragment the packet and reports it back, which shows up as M. If that ICMP message is filtered, you get dots instead. To find the exact limit, sweep with DF set (say 1400 to 1500 in steps of 10) and watch where ! turns into M or .. See MTU and fragmentation for why this bites tunnels.

Testing the return path with a source address

R1 has a LAN on GigabitEthernet0/1 (192.168.1.0/24) and a WAN link to R2 on GigabitEthernet0/0 (10.1.12.0/30). R2 connects to R3, whose loopback is 10.3.3.3. LAN users cannot reach it. On R1:

R1# ping 10.3.3.3
!!!!!
Success rate is 100 percent (5/5)
R1# ping 10.3.3.3 source GigabitEthernet0/1
Packet sent with a source address of 192.168.1.1
.....
Success rate is 0 percent (0/5)

The first ping was sourced from 10.1.12.1, and R3 already has a route to the WAN links. The second uses the same forward path, so the requests still arrive. The replies to 192.168.1.1 die because nobody advertised the LAN:

R3# show ip route 192.168.1.1
% Network not in table

The fault is on the far side. The routing decision process page covers how R3 picks (or fails to pick) a route for that reply.

Traceroute

R1# traceroute 10.3.3.3
Type escape sequence to abort.
Tracing the route to 10.3.3.3

  1 10.1.12.2 4 msec 4 msec 4 msec
  2 10.1.23.3 8 msec 8 msec *

How IOS traceroute works. It sends UDP datagrams to an unlikely port (33434 by default). The first three probes have TTL 1. The first router decrements TTL to zero, drops them, and returns ICMP Time Exceeded, which reveals hop 1. Three probes with TTL 2 reveal hop 2, and so on up to a maximum TTL of 30. At the destination nothing listens on that port, so it returns ICMP Port Unreachable and the trace ends.

Windows tracert uses the same TTL trick but sends ICMP Echo Requests instead of UDP, so a firewall can pass one and block the other.

Each hop shows one result per probe. Cisco documents these markers:

On screen the letters usually appear with an exclamation mark in front, such as !H or !A. Rows of * * * down to hop 30 mean probes leave but nothing answers: often a missing return route or a filter. Press the escape sequence (Ctrl+Shift+6) to stop it.

Type traceroute alone for the extended version: Source address, Numeric display [n], Timeout in seconds [3], Probe count [3], Minimum Time to Live [1], Maximum Time to Live [30] and Port Number [33434]. Or on one line:

R1# traceroute 10.3.3.3 source Loopback0

A troubleshooting workflow

Inside the bigger troubleshooting methodology, work bottom up:

1. Interfaces: show ip interface brief. You want up/up and the right IP. Anything else is a Layer 1 or 2 problem first.
2. Neighbors: show arp. An Incomplete entry for the next hop means ARP is not getting answered.
3. Forward route: show ip route <destination>. No match and no default route means R1 drops the packet.
4. Two pings: one plain, one sourced from the user-facing interface. If only the sourced one fails, look at the return path.
5. Sourced traceroute: find the last hop that answers, log into the next device, and repeat steps 1 to 3 there.

Common mistakes

1. Pinging from the wrong source and calling the far side fine. A plain ping uses the outgoing interface address, which the far side knows, while the users’ subnet has no return route. Always repeat the test from the LAN interface or loopback the real traffic uses.

2. Panicking over .!!!!. One dot on the first attempt is usually ARP. Ping again before changing anything.

3. Treating dots as proof the target is down. The host may be up and filtering ICMP, or the reply may be taking a broken path back.

4. Testing MTU without the DF bit. Without df-bit, a router can fragment the packet, the ping succeeds, and the small link stays hidden.

Lab to try tonight

1. Build R1, R2 and R3 in a line with a loopback on each. Use static routes so R1 reaches R3’s loopback, but leave out R3’s route back to R1’s loopback.
2. From R1, ping R3’s loopback. Note the source and the result.
3. Ping it again with source Loopback0, then traceroute with the same source. Watch both fail.
4. Add the missing route on R3 and repeat.
5. Set ip mtu 1400 on the R2 to R3 link, then run ping <R3 loopback> size 1500 df-bit and a DF sweep from 1380 to 1500.
6. Run clear arp-cache on R1, ping R2, and watch for the leading dot.

Cheat strip

Frequently asked questions

Q: Why does my first ping on a Cisco router show .!!!!? A: The router had no ARP entry for the next hop, so it dropped the first echo while ARP resolved. Ping again and it should be !!!!!.

Q: Does IOS traceroute use ICMP or UDP? A: IOS sends UDP probes to port 33434 by default and listens for ICMP Time Exceeded from each hop and Port Unreachable from the destination. Windows tracert sends ICMP Echo Requests.

Q: What does an M in ping output mean? A: Could not fragment. The DF bit was set and the packet was bigger than the MTU of a link along the path.

Q: Why does ping work from the router but not from a PC on its LAN? A: The router’s default source is its outgoing interface, which the far side already knows. The PC’s subnet probably has no route back. Prove it with ping <target> source <LAN interface>.