Troubleshoot a DMVPN phase 3 architecture

In the last article, I explained how to configure DMVPN phase3, but what are the most useful commands to troubleshoot this type of network architecture?

Five are the main group of commands used to troubleshoot a DMVPN topology:

• show dmvpn […]
• show ip nhrp […]
• show ip eigrp […]
• show crypto […]

The “show dmvpn” and “show ip nhrp” commands permit to obtain the state of the tunnels.

On hub router, all tunnels are dynamic (D attribute) because it waits the registration from spokes routers (“ip nhrp map multicast dynamic”).

Ciscozine#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel1, IPv4 NHRP Details
Type:Hub, NHRP Peers:2,

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 27.27.27.2             10.0.1.2    UP 00:28:32     D
     1 37.37.37.3             10.0.1.3    UP 00:28:42     D

Ciscozine#

By default, the spoke routers have only static tunnel (S attribute) to the hub router (“ip nhrp map 10.0.1.1 17.17.17.1”).

Spoke2#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel1, IPv4 NHRP Details
Type:Spoke, NHRP Peers:1,

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 17.17.17.1             10.0.1.1    UP 00:27:26     S

Spoke2#

The static tunnel can also be checked by the “show ip nhrp” command:

Spoke2#show ip nhrp
10.0.1.1/32 via 10.0.1.1
   Tunnel1 created 02:11:15, never expire
   Type: static, Flags: used
   NBMA address: 17.17.17.1
Spoke2#

Note: Due to the static nature, the tunnel “never expire”.

What happens when a spoke tries to reach another spoke? Dynamic tunnel are established.

Suppose Spoke2 (192.168.2.2) wants to reach Spoke3 (192.168.3.3). The first packet reaches the hub, then the Spoke3 router:

Spoke2#traceroute 192.168.3.3 source ethernet 0/0
Type escape sequence to abort.
Tracing the route to 192.168.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 10.0.1.1 5 msec 4 msec 2 msec
  2 10.0.1.3 2 msec 5 msec 5 msec
Spoke2#

Then the hub “link” this traffic back onto the DMVPN network, triggering the NHRP process on HUB to generate the traffic indication to Spoke2 to resolve a better next hop for the remote network 192.168.3.0 (Spoke3).

At this point, the spokes can modify their routing table entries to reflect the NHRP shortcut route and use it to reach the remote spoke.

Spoke2#traceroute 192.168.3.3 source ethernet 0/0
Type escape sequence to abort.
Tracing the route to 192.168.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 10.0.1.3 5 msec 5 msec 8 msec
Spoke2#

As you can see, the new routes are learned by the NHRP process and not by the EIGRP process (notice the “H” and not the “D” flag):

Spoke2#show ip route eigrp
!omitted!
D     192.168.0.0/16 [90/26905600] via 10.0.1.1, 01:11:06, Tunnel1
Spoke2#

Spoke2#show ip route nhrp
!omitted!
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
H        10.0.1.3/32 is directly connected, 00:00:01, Tunnel1
H     192.168.3.0/24 [250/1] via 10.0.1.3, 00:00:01, Tunnel1
Spoke2#

and Dynamic tunnel are established:

Spoke2#show dmvpn detail
!omitted!
# Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb    Target Network
----- --------------- --------------- ----- -------- ----- -----------------
    1 17.17.17.1             10.0.1.1    UP 01:13:37    S        10.0.1.1/32
    2 37.37.37.3             10.0.1.3    UP 00:00:06  DT1        10.0.1.3/32
      37.37.37.3             10.0.1.3    UP 00:00:06  DT1     192.168.3.0/24
    1 27.27.27.2             10.0.1.2    UP 00:00:06  DLX     192.168.2.0/24
!omitted!
Spoke2#

Note: DT1 attribute means that a dynamic (D) tunnel is established and the route is installed (T1).

The second line is a Spoke3’s tunnel, the third line is an entry for the subnet behind Spoke3 and the last one it’s formed for the local subnet.

A similar output can be verified using:

Spoke2#show ip nhrp
10.0.1.1/32 via 10.0.1.1
   Tunnel1 created 02:12:19, never expire
   Type: static, Flags: used
   NBMA address: 17.17.17.1
10.0.1.3/32 via 10.0.1.3
   Tunnel1 created 00:00:16, expire 00:04:43
   Type: dynamic, Flags: router nhop rib
   NBMA address: 37.37.37.3
192.168.2.0/24 via 10.0.1.2
   Tunnel1 created 00:00:16, expire 00:04:43
   Type: dynamic, Flags: router unique local
   NBMA address: 27.27.27.2
    (no-socket)
192.168.3.0/24 via 10.0.1.3
   Tunnel1 created 00:00:16, expire 00:04:43
   Type: dynamic, Flags: router rib
   NBMA address: 37.37.37.3
Spoke2#

Obviously, if the dynamic tunnel between two spokes are no more used, a counter will tear down the tunnels. This is defined by the command “ip nhrp holdtime”.

Due to its static nature, on the hub tunnels are unchanged:

Ciscozine#show ip nhrp
10.0.1.2/32 via 10.0.1.2
   Tunnel1 created 00:00:14, expire 00:04:45
   Type: dynamic, Flags: unique registered used nhop
   NBMA address: 27.27.27.2
10.0.1.3/32 via 10.0.1.3
   Tunnel1 created 00:00:23, expire 00:04:36
   Type: dynamic, Flags: unique registered used nhop
   NBMA address: 37.37.37.3
Ciscozine#

By default, each 100 seconds a spoke sends NHRP registration requests to the hub. This value can be customized using the command “ip nhrp registration timeout”. The default value is one third NHRP holdtime value.

To find more info about routing process, two show commands can be used: “show ip eigrp neighbors” and “show ip route eigrp”.

The “show ip eigrp neighbors” displays EIGRP neighborship informations; in a DMVPN environment, the only eigrp neighborship are between hub and spoke and never between spoke to spoke, also when a spoke to spoke relationship is established.

Ciscozine#show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(100)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
1   10.0.1.2                Tu1                      14 00:43:37    7  1440  0  6
0   10.0.1.3                Tu1                      14 00:43:46   15  1440  0  7
Ciscozine#

Spoke2#show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(100)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
0   10.0.1.1                Tu1                      13 00:43:12   18  1398  0  10
Spoke2#

The “show ip route eigrp” shows route learned by EIGRP. In the hub router, there are all eigrp routes: the first one is made by the ” ip summary-address eigrp 100 192.168.0.0 255.255.0.0″ command under the hub tunnel interface, while the others are learned by the two spokes.

Ciscozine#show ip route eigrp
!omitted!
D     192.168.0.0/16 is a summary, 00:33:54, Null0
D     192.168.2.0/24 [90/26905600] via 10.0.1.2, 00:33:41, Tunnel1
D     192.168.3.0/24 [90/26905600] via 10.0.1.3, 00:33:50, Tunnel1
Ciscozine#

On spoke routers, only one route is learned by the hub: the summary route.

Spoke2#show ip route eigrp
!omitted!
D     192.168.0.0/16 [90/26905600] via 10.0.1.1, 00:31:42, Tunnel1
Spoke2#

Remember: During spoke to spoke relationship, the spoke networks are learned by NHRP (H) process.

And what about IPSEC troubleshooting commands?

Mainly there are 2 commands:

• show crypto isakmp sa: shows the Internet Security Association Management Protocol (ISAKMP) security associations (SAs) built between peers (Phase I)

  Ciscozine#show crypto isa sa
  IPv4 Crypto ISAKMP SA
  dst             src             state          conn-id status
  17.17.17.1      37.37.37.3      QM_IDLE           1001 ACTIVE
  17.17.17.1      27.27.27.2      QM_IDLE           1002 ACTIVE
  
  IPv6 Crypto ISAKMP SA
  
  Ciscozine#
  
  

• show crypto ipsec sa: shows IPsec Security associations built between peers and the number of encrypted/decrypted packets per tunnel.

  Ciscozine#show crypto ipsec sa
  
  interface: Tunnel1
  Crypto map tag: Tunnel1-head-0, local addr 17.17.17.1
  
  protected vrf: (none)
  local ident (addr/mask/prot/port): (17.17.17.1/255.255.255.255/47/0)
  remote ident (addr/mask/prot/port): (27.27.27.2/255.255.255.255/47/0)
  current_peer 27.27.27.2 port 500
  PERMIT, flags={origin_is_acl,}
  #pkts encaps: 14, #pkts encrypt: 14, #pkts digest: 14
  #pkts decaps: 14, #pkts decrypt: 14, #pkts verify: 14
  #pkts compressed: 0, #pkts decompressed: 0
  #pkts not compressed: 0, #pkts compr. failed: 0
  #pkts not decompressed: 0, #pkts decompress failed: 0
  #send errors 0, #recv errors 0
  
  local crypto endpt.: 17.17.17.1, remote crypto endpt.: 27.27.27.2
  plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb (none)
  current outbound spi: 0x8B504438(2337293368)
  PFS (Y/N): N, DH group: none
  
  inbound esp sas:
  spi: 0xC6042A88(3322161800)
  transform: esp-256-aes esp-sha256-hmac ,
  in use settings ={Tunnel, }
  conn id: 3, flow_id: SW:3, sibling_flags 80000040, crypto map: Tunnel1-head-0
  sa timing: remaining key lifetime (k/sec): (4211725/3565)
  IV size: 16 bytes
  replay detection support: Y
  ecn bit support: Y status: off
  Status: ACTIVE(ACTIVE)
  
  inbound ah sas:
  
  inbound pcp sas:
  
  outbound esp sas:
  spi: 0x8B504438(2337293368)
  transform: esp-256-aes esp-sha256-hmac ,
  in use settings ={Tunnel, }
  conn id: 4, flow_id: SW:4, sibling_flags 80000040, crypto map: Tunnel1-head-0
  sa timing: remaining key lifetime (k/sec): (4211726/3565)
  IV size: 16 bytes
  replay detection support: Y
  ecn bit support: Y status: off
  Status: ACTIVE(ACTIVE)
  
  outbound ah sas:
  
  outbound pcp sas:
  
  protected vrf: (none)
  local ident (addr/mask/prot/port): (17.17.17.1/255.255.255.255/47/0)
  remote ident (addr/mask/prot/port): (37.37.37.3/255.255.255.255/47/0)
  current_peer 37.37.37.3 port 500
  PERMIT, flags={origin_is_acl,}
  #pkts encaps: 18, #pkts encrypt: 18, #pkts digest: 18
  #pkts decaps: 17, #pkts decrypt: 17, #pkts verify: 17
  #pkts compressed: 0, #pkts decompressed: 0
  #pkts not compressed: 0, #pkts compr. failed: 0
  #pkts not decompressed: 0, #pkts decompress failed: 0
  #send errors 0, #recv errors 0
  
  local crypto endpt.: 17.17.17.1, remote crypto endpt.: 37.37.37.3
  plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb (none)
  current outbound spi: 0xF01B8F07(4028337927)
  PFS (Y/N): N, DH group: none
  
  inbound esp sas:
  spi: 0x2F740B30(796134192)
  transform: esp-256-aes esp-sha256-hmac ,
  in use settings ={Tunnel, }
  conn id: 1, flow_id: SW:1, sibling_flags 80000040, crypto map: Tunnel1-head-0
  sa timing: remaining key lifetime (k/sec): (4245042/3556)
  IV size: 16 bytes
  replay detection support: Y
  ecn bit support: Y status: off
  Status: ACTIVE(ACTIVE)
  
  inbound ah sas:
  
  inbound pcp sas:
  
  outbound esp sas:
  spi: 0xF01B8F07(4028337927)
  transform: esp-256-aes esp-sha256-hmac ,
  in use settings ={Tunnel, }
  conn id: 2, flow_id: SW:2, sibling_flags 80000040, crypto map: Tunnel1-head-0
  sa timing: remaining key lifetime (k/sec): (4245042/3556)
  IV size: 16 bytes
  replay detection support: Y
  ecn bit support: Y status: off
  Status: ACTIVE(ACTIVE)
  
  outbound ah sas:
  
  outbound pcp sas:
  Ciscozine#

Note: Obviously, all the show commands can be used to analyze DMVPN Phase1 and Phase2, but the output can change slightly due to the different architectures.