Recently, I needed to send SPAN traffic to a network analyzer, but I had to resolve two problems:
- The network analyzer was installed on a virtual machine.
- I could not manage the virtual switch.
For these reasons, I decided to send ERSPAN (Encapsulated Remote Switched Port Analyze) traffic directly to the virtual machine but a problem still persisted. How can I remove ERSPAN header, if the software used is not able to decode the tunnel?
First of all, you need to find which ERSPAN version is used. To do it, open a wireshark / tcpdump on your virtual machine where the network analyzer software is installed and check the ERSPAN Version field as below:
There are two versions:
- Version 1 named as “type II” (this case)
- Version 2 named as “type III”
Below the ethernet packet of a ERSPAN type II session:
As you notice, a lot of overhead is added by GRE / ERSPAN protocol.
In that case, the ERSPAN configuration is defined on a Cisco Catalyst 9500; my goal is to mirror vlan 2020 IP traffic to the network analyzer (10.0.50.103).
Ciscozine(config)#monitor session 10 type erspan-source Ciscozine(config-mon-erspan-src)# source vlan 2020 Ciscozine(config-mon-erspan-src)# no shutdown Ciscozine(config-mon-erspan-src)# destination Ciscozine(config-mon-erspan-src-dst)# erspan-id 10 Ciscozine(config-mon-erspan-src-dst)# mtu 1900 NOTE: Configured MTU of size 1900 will be part of ERSPAN payload and captured at dest port, rest will be discarded Ciscozine(config-mon-erspan-src-dst)# ip address 10.0.50.103 Ciscozine(config-mon-erspan-src-dst)# origin ip address 10.0.50.1
Remember: Make sure you increase the MTU of the ERSPAN session and the IP path to accommodate the extra bytes of tunnel header.
Ciscozine#sh monitor session 10 Session 10 Type : ERSPAN Source Session Status : Admin Enabled Source VLANs : Both : 2020 Destination IP Address : 10.0.50.103 MTU : 1900 Destination ERSPAN ID : 10 Origin IP Address : 10.0.50.1 Ciscozine#
Note: not all devices support MTU truncation, like in that case. If your devices is IOS-XE and you want to find that information use the command:
Ciscozine#show capability feature monitor erspan-source ERSPAN Source Session:ERSPAN Source Session Supported: TRUE No of Rx ERSPAN source session: 8 No of Tx ERSPAN source session: 8 ERSPAN Header Type supported: II and III ACL filter Supported: TRUE SGT filter Supported: TRUE Fragmentation Supported: TRUE Truncation Supported: FALSE Sequence number Supported: FALSE QOS Supported: TRUE Ciscozine#
In this POC, an Ubuntu 20.04 with two virtual NIC is used:
- ens160: the management server interface (192.168.100.28)
- ens192: the erspan destination (IP 10.0.50.103)
Increase ethernet MTU
ifconfig ens192 mtu 1900
Create a new virtual interface
ip link add dev cisco_erspan mtu 1900 type erspan seq key 10 local 10.0.50.103 remote 10.0.50.1 erspan_ver 1
- The new interface “cisco_erspan” decapsulates the GRE / ERSPAN tunnel
- The key must be equal to the “erspan-id” defined in the ERSPAN switch configuration . In that case the erspan-id is “10”, so the key must be “10”.
- The local IP is the ens192 address (the IP address of the virtual machine).
- The remote IP is the Catalyst 9500 address.
- The ERSPAN version is 1 (type II).
Enable the new virtual interface
ip link set cisco_erspan up
If you try to check cisco_erspan statistics, you notice that RX packets counters do not increase:
cisco_erspan: flags=4163 mtu 1900
inet6 fe80::9cd6:77ff:fe47:c460 prefixlen 64 scopeid 0x20
ether 9e:d6:77:47:c4:60 txqueuelen 1000 (Ethernet)
RX packets 1 bytes 58 (58.0 B)
RX errors 19342 dropped 0 overruns 19342 frame 0
TX packets 21 bytes 2453 (2.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
to get more details:
root@Ciscozine-Analyzer:/home/ciscozine# ip -d -s -s link show dev cisco_erspan 10: cisco_erspan@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1900 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 9e:d6:77:47:c4:60 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 0 erspan remote 10.0.50.1 local 10.0.50.103 ttl inherit ikey 0.0.0.10 okey 0.0.0.10 iseq oseq erspan_index 0 erspan_ver 1 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 RX: bytes packets errors dropped overrun mcast 58 1 173262 0 0 0 RX errors: length crc frame fifo missed 0 0 0 173262 0 TX: bytes packets errors dropped carrier collsns 2954 27 0 0 0 0 TX errors: aborted fifo window heartbeat transns 0 0 0 0 0 root@Ciscozine-Analyzer:/home/ciscozine#
The fifo errors are caused by input sequence number mismatch. In fact, when ERSPAN tunnel receives mirrored packet, it checks whether the mirrored packets are in sequence (based on the sequence number in the erspan header). If it’s not, then kernel will drop it.
To avoid the dropping
ip link set dev cisco_erspan type erspan noiseq
The rx errors stop to increase, the RX packets counters begin to increase and the network analyzer starts to analyze IP packets!