Jump to:navigation, search
Wiki






























De.png
En.png
Fr.png






Securepoint UTM error analysis with tcpdump
Last adaption: 02.2024
New:
  • Layout update
notempty
This article refers to a Resellerpreview

11.7


Introduction

If a TCP/IP connection does not work, the firewall offers several ways to track down the errors. Specifically, there are three levels of analysis that descend deeper and deeper into the system. The top level can be accessed via the web interface, the lower levels require a text console, either via SSH or locally on the device.


The levels of error analysis

Level 1: Livelog

The live log can be found under Log .

In addition to application messages, the Livelog also displays messages from the packet filter. By default, however, you can only see if the default policy rejects packets for which there is no suitable firewall rule. However, logging can be configured for each firewall rule so that an entry also appears if this rule takes effect.

This results in three options for troubleshooting:

  1. The packet you are looking for appears and is discarded (DROP)
  2. The packet you are looking for appears and is accepted (ACCEPT)
  3. The packet you are looking for does not appear


This allows the cause of the fault to be localized:

  1. If a packet is discarded, the appropriate FW rule is missing, it has not been created correctly or is not yet effective (rules have not yet been updated)
  2. If a packet is accepted, then the error is probably in the direction of the destination host.
  3. If no packet is visible in the live log, then presumably none is arriving at the firewall. The error here is probably in the direction of the source host.

Level 2: CLI

The CLI (Command Line Interface) is the interface to the firewall server, which is used by the web interface and can also be used on the console for manual configuration of the firewall. However, the CLI is only of limited use for troubleshooting connection problems.

The CLI can be accessed via the web interface or via a text console. This requires a user with administration rights.


Level 3: Linux-Shell

The Linux shell accesses the underlying operating system. It provides access to many system parameters that are not accessible via the web interface or the CLI - albeit mostly read-only. With the packet sniffer tcpdump, a very powerful tool for traffic analysis is available on the linux shell.

The Linux shell can only be accessed with the help of a user who has root authorization.



Tcpdump on the root console

Requirements

The requirements for using tcpdump are as follows:

  • A "root" user in the administrator group
  • An SSH client (e.g. PuTTY) or a local console on the firewall

Parameters for control and filtering

Using tcpdump without parameters would simply display EVERY tcp/ip packet on the console. It is therefore necessary to specify a search pattern using appropriate filter parameters.

-i defines the interface on which incoming or outgoing packets are to be displayed.
Example:
tcpdump -i A1


proto defines the protocol number at transport level.
Examples:
tcpdump -i A1 proto 1 shows ICMP-packets
tcpdump -i A1 proto 6 shows TCP-packets
tcpdump -i A1 proto 17 shows UDP-packets
tcpdump -i A1 proto 50 shows ESP-packets


port defines tcp or udp ports.
Example: tcpdump -i A1 port 80


host defines a specific host with its IP as the source or destination.
Example: tcpdump -i A1 host 10.0.0.1


net defines a network (source or destination)
Example: tcpdump -i A1 net 10.0.0.0/24


Filter parameters can also be linked to logical operators. In this way, all ICMP packets for host 10.0.0.10 can be displayed:
tcpdump -i A1 proto 1 and host 10.0.0.10


Further control parameters are for example: -n prevents tcpdump from attempting to perform a reverse lookup on IP addresses in order to display host names.

-s defines the length up to which a packet is recorded. With the value 0, the entire packet is recorded.

-w redirects the output to a text file. This can then be further examined with an analysis tool, e.g. wireshark. The path of the text file is specified as the value, e.g. /var/tcpdump.txt



The traffic analysis

In our example, we have two networks of the head office and a branch location of a company whose gateways connect these networks with each other via an IPSec tunnel. The head office has the internal network 10.0.0.0/24 and the external address 198.51.100.75. The branch office has the internal network 10.4.0.0/24 and the external address 198.51.100.4. The host 10.0.0.10 tries to reach the host 10.4.0.10, but fails.


Initiator site: Internal

We start our analysis on the internal interface of the gateway of the control center. To do this, we log in there via ssh as user "root" and execute the following command:
tcpdump -i A1 proto 1 -n

The parameter proto 1 refers to the protocol number 1 of the ICMP protocol at transport level. Alternatively, we can also search for packets destined for the remote subnet:
tcpdump -i A1 net 10.4.0.0/24 -n

If we now try to ping, we get the following result:
09:59:44.445502 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 1, length 40 09:59:49.847558 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 2, length 40 09:59:54.855574 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 3, length 40 09:59:59.861512 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 4, length 40


We see that the requests arrive at the internal interface of the central firewall. However, there is no response.


Initiator site: External

The outgoing interface of the central firewall must be viewed for further tracking. However, there is a problem here: as we can only view the transport of the transmitted data in encrypted form at this point, the ICMP packet remains hidden from us in plain text - and that is how it should be. However, we can try to correlate incoming plain text and outgoing encrypted packets. To do this, we start tcpdump with the following parameters:
tcpdump -i any proto 1 or proto 50 -n

We search for ICMP packets or encrypted packets at any interface. The ESP protocol used by IPSec for this purpose has the protocol number 50 at the transport level. If there are several IPSec connections on the gateway, it is also possible to search for packets for the remote subnet or the remote gateway:
tcpdump -i any net 10.4.0.0/24 or host 198.51.100.4 -n

We get the following result:
10:21:39.710743 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 1, length 40 10:21:39.710799 IP 198.51.100.75 > 198.51.100.4: ESP(spi=0xc155682b,seq=0x9), length 92 10:21:45.056141 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 2, length 40 10:21:45.056235 IP 198.51.100.75 > 198.51.100.4: ESP(spi=0xc155682b,seq=0xa), length 92 10:21:50.065278 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 3, length 40 10:21:50.065332 IP 198.51.100.75 > 198.51.100.4: ESP(spi=0xc155682b,seq=0xb), length 92 10:21:55.075902 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 4, length 40 10:21:55.075947 IP 198.51.100.75 > 198.51.100.4: ESP(spi=0xc155682b,seq=0xc), length 92

We see four ICMP echo request packets with consecutive sequence numbers, each directly followed by an ESP packet. These ESP packets also have consecutive sequence numbers. This means that on the central firewall side, all data packets for the remote subnet are encrypted and sent to the remote gateway.

However, we may get the following result:

10:58:15.436094 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 1, length 40 10:58:15.436127 IP 198.51.100.75 > 10.4.0.10: ICMP echo request, id 512, seq 1, length 40 10:58:20.810201 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 2, length 40 10:58:20.810230 IP 198.51.100.75 > 10.4.0.10: ICMP echo request, id 512, seq 2, length 40 10:58:25.820479 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 3, length 40 10:58:25.820533 IP 198.51.100.75 > 10.4.0.10: ICMP echo request, id 512, seq 3, length 40 10:58:30.830402 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 4, length 40 10:58:30.830470 IP 198.51.100.75 > 10.4.0.10: ICMP echo request, id 512, seq 4, length 40

We see each packet twice, recognizable by the same sequence number. However, the second version of the packet has the IP of the external interface as its source. A HideNAT was therefore performed via the external interface. In this case, we have probably forgotten to set the HideNAT exception within the port filter rule from our own subnet to the remote subnet or to activate the "No NAT for IPSec connections" option in the implicit rules. As a result, instead of going encrypted into the IPSec tunnel, the packet is NATed and sent towards the default gateway.


Responder site: External

If we assume that everything is working on the gateway at the head office, we need to look further on the gateway at the branch office. First, we make sure that the encrypted data also arrives at the remote station. The fact that it is sent from the head office does not necessarily mean that it arrives! We therefore execute the following command on the SSH console at the branch office:
tcpdump -i A0 proto 50 -n

The result is the following output:
11:24:55.833742 IP 198.51.100.75 > 198.51.100.4: ESP(spi=0xc155682b,seq=0x45), length 92 11:25:00.890782 IP 198.51.100.75 > 198.51.100.4: ESP(spi=0xc155682b,seq=0x46), length 92 11:25:05.899056 IP 198.51.100.75 > 198.51.100.4: ESP(spi=0xc155682b,seq=0x47), length 92 11:25:10.908124 IP 198.51.100.75 > 198.51.100.4: ESP(spi=0xc155682b,seq=0x48), length 92


Responder site: Internal

We can therefore be sure that the encrypted packets also arrive at the store gateway! The last stage is now the internal interface of the store gateway. Here we check whether the ICMP packet is also sent to the internal network in the direction of the target host:
tcpdump -i A1 proto 1 -n

If everything is also configured correctly on the branch gateway with regard to port filter rules, you should see the following output:
11:32:11.592831 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 1, length 40 11:32:16.733971 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 2, length 40 11:32:21.742500 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 3, length 40 11:32:26.753739 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 4, length 40

From this we can clearly see that the target host is at least online and physically accessible. Why can we be so sure of this? This becomes clear when we look at the output that we receive when we search for packets from the target host instead of the ICMP protocol:
root@standort-4:~# tcpdump -i A1 host 10.4.0.10 -n 11:44:39.553889 ARP, Request who-has 10.4.0.10 tell 10.4.0.1, length 28 11:44:39.554212 ARP, Reply 10.4.0.10 is-at 08:00:27:e1:fd:ab, length 46 11:44:39.672145 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 1, length 40 11:44:44.682827 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 2, length 40 11:44:49.692773 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 3, length 40 11:44:49.699543 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 4, length 40

We can see here that before our pings, the gateway sends an ARP request to the internal network to determine the MAC address of the target host. This is followed by a response from the target host in which it communicates this. Only then can IP packets be transmitted to the target host. The results of ARP requests are temporarily stored in the neighbor table, also known as the ARP cache, so that a new ARP request does not have to be made for every IP packet. We can view this with the following command:

ip n

Among other things, we find an entry in the output that belongs to our target host:

10.4.0.10 dev A1 lladdr 08:00:27:e1:fd:ab REACHABLE

This clearly proves that the target host is online and physically accessible. If it were not, the Neighbor table would contain the following entry:

10.4.0.10 dev A1 FAILED

If the gateway at least attempts to deliver a packet to the target host, the error is no longer within the tunnel or firewall configuration, but clearly at the target host. Either we do not see any IP packets, but an entry in the Neighbor table, which indicates that the target host is physically unreachable. Or we see an IP packet and can conclude from this that the target host is physically reachable but is not responding to the connection request for some reason.


Possible sources of error on the target host

Once it has been established that the error can only be on the target host, it should also be found there. However, we can also narrow down the error further on the gateway on the responder side if we look at whether and which packets come back from the target host:

tcpdump -i A1 host 10.4.0.10 -n

If only the outgoing pings from the NextGen UTM are visible, then it can be assumed that the target host is not responding due to an active local firewall:

11:32:11.592831 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 1, length 40 11:32:16.733971 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 2, length 40 11:32:21.742500 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 3, length 40 11:32:26.753739 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 4, length 40

If the following error occurs, an incorrectly set subnet mask in the network configuration of the target host can be assumed:

11:32:11.592831 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 1, length 40 11:32:39.553889 ARP, Request who-has 10.0.0.10 tell 10.4.0.10, length 28 11:32:16.733971 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 2, length 40 11:32:39.553889 ARP, Request who-has 10.0.0.10 tell 10.4.0.10, length 28 11:32:21.742500 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 3, length 40 11:32:39.553889 ARP, Request who-has 10.0.0.10 tell 10.4.0.10, length 28 11:32:26.753739 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 4, length 40 11:32:39.553889 ARP, Request who-has 10.0.0.10 tell 10.4.0.10, length 28

The target host attempts to determine the MAC address of the source host with an ARP request as if it were in the same subnet. This clearly indicates an incorrect subnet mask. Even an incorrectly set default gateway can be detected in this way:

11:32:11.592831 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 1, length 40 11:32:39.553889 ARP, Request who-has 10.4.0.254 tell 10.4.0.10, length 28 11:32:16.733971 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 2, length 40 11:32:39.553889 ARP, Request who-has 10.4.0.254 tell 10.4.0.10, length 28 11:32:21.742500 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 3, length 40 11:32:39.553889 ARP, Request who-has 10.4.0.254 tell 10.4.0.10, length 28 11:32:26.753739 IP 10.0.0.10 > 10.4.0.10: ICMP echo request, id 512, seq 4, length 40 11:32:39.553889 ARP, Request who-has 10.4.0.254 tell 10.4.0.10, length 28



Further diagnostic options for tcp connections

In addition to the mere accessibility of a host, other aspects of a connection can be examined, such as the correct establishment of the TCP 3-way handshake. If the RST flag is set in the response to the initial packet (SYN flag), the target host is reached, but the corresponding service is not accessible:

root@standort-4:~# tcpdump -i A1 port 3389 -n 14:24:14.433460 IP 10.0.0.10.50795 > 10.4.0.10.3389: Flags [S], seq 315479174, win 64240, options [mss 1341,nop,wscale 8,nop,nop,sackOK], length 0 14:24:14.433725 IP 10.4.0.10.3389 > 10.0.0.10.50795: Flags [R.], seq 0, ack 315479175, win 0, length 0

Even more in-depth analyses of the data stream can be carried out if it is recorded in its entirety and examined in an analysis tool such as Wireshark:

root@standort-4:~# tcpdump -i A1 port 25 -w /tmp/dump.txt -s 0 -n tcpdump: listening on A1, link-type EN10MB (Ethernet), capture size 262144 bytes 12 packets captured 12 packets received by filter 0 packets dropped by kernel