New Technology for detecting operating system fingerprints of remote hosts

Author: too2y

I. Introduction

In recent years, the network security evaluation software has been gradually accepted by the network security community and quickly spread in a large scope. As part of the evaluation software, remote operating system detection must meet the following conditions:
-Accuracy: avoid incorrect detection results;
-Impact of firewalls and intrusion detection systems: avoid affecting (or being affected) firewalls and intrusion detection systems;
-Elegant: low network traffic and non-hazardous segmentation;
-Smart: easy to expand and identify databases and automatic detection functions;
-Speed: High-Speed fingerprint detection tools should allow a wide range of network scans.
I will introduce you to a new operating system fingerprint detection tool-ring. In some cases, other tools may fail, but the good features and accurate results of the ring will surely satisfy you.
Ring is an open-source software designed for experimental concepts and testing purposes.

Ii. Technology Development Level

2.1 Brief History of operating system detection technology
-This technique is used in many test tools to obtain the identification information of certain services. It is often achieved through the collection and analysis of binary files.
-TCP segmentation (Standard/non-standard) response analysis depends on the different responses of different operating systems to specific segments. Popular tools include Savage's queso and Fyodor's NMAP. They all use many variants from this technology.
-ICMP response analysis is a new technology. It sends UPD or ICMP request packets and analyzes various ICMP responses. The X-probe of Ofir Arkin is the technology used. In general, X-probe works better, but when the firewall blocks some protocols, the results are not satisfactory.
-The initialization serial number (ISN) analysis randomly generates different exploits in the TCP stack. the operating system of the remote host is determined by identifying sufficient test results. (For more information, see Maid, M. (Apr, 2001), strange attractors and TCP/IP Sequence Number Analysis)
-Special dos can also be used for operating system fingerprint detection, not just by hackers. In some special cases, denial of service can detect accurate results.

2.2 details of stack Query Technology
Stack query technology detects the system by measuring the response of the remote host TCP/IP stack to different requests. Most operating systems will respond to special segment requests in a special way.
NMAP and queso are based on this technology. They generate a set of open (not open) ports that send TCP and UDP requests to the remote target host. In this case, the useful response information of the remote host is received by the probe tool and analyzed.
These technologies usually enable the Security Evaluation Software to get some information about the type and version at a low latency.
In several ways, this technology is so accurate:
-Each operating system (even if a patch is installed) is usually implemented using its own IP stack.
-TCP/IP specifications are not strictly enforced. Each different implementation will have its own characteristics, which makes it possible for a successful test.
-Specifications may be disrupted, some selective features may be used, and some other systems may not.
-Some private improvements to the IP protocol may also be implemented, which becomes a feature of some operating systems.

2.3 restrictions on typical tools
NMAP can recognize 500 different operating systems, but the premise is the stability of the network environment, the target host must open a TCP port, a closed TCP port and a closed UDP port. If the above conditions are not met, the accuracy of the test results will be greatly reduced.
For the sake of security, the current network system usually opens only one visible TCP port, while the datagram received by other ports is filtered out by the firewall or packet filtering device.
In such a network environment protected by the firewall, the UDP port closed based on the ICMP protocol and the TCP port detection tool closed, such as NMAP and xprobe, work poorly.

Iii. Use of TCP/IP protocol

As a data transmission protocol, TCP is built on the IP Protocol. Its definition can be found on rfc793. TCP/IP is the main network protocol used on the Internet.
The success of TCP lies in its reliability: error detection and management, control of data flow and congestion, and retransmission mechanism.
To meet this requirement, TCP becomes a connection-oriented protocol. Its operating mechanism is as follows:
1. Establish a connection
2. Data Transmission
3. Terminate the connection
TCP is built on the IP protocol. In the case of network congestion or routing problems, the IP protocol cannot provide reliable end-to-end data transmission. In addition, IP is a non-connection protocol, so connection control is established on the TCP layer.
TCP provides multiple control over the connection in its header. Serial numbers and validation numbers are used to better manage the re-transmission of data packets and control various special error conditions. The URG, ack, Psh, RST, Syn, and fin fields in the TCP datagram header are set to manage the TCP connection status. For more information, see rfc793.
In order to better understand the principle of the ring in the future, the following lists the three handshakes for TCP connection establishment: (the client actively initiates a connection to the server)

Client Server
│ -------- SYN ---------> │
│ SYN-ACK ------- │
│ -------- Ack ---------> │

During the network transmission of a datagram, some segments may be lost during transmission. Theoretically, each datagram must be confirmed by the receiver. TCP maintains a list of confirmed data packets.
Note: If a datagram is not received within the expected time, it will be treated as a loss.

Moreover, TCP automatically processes the real sequence of received data packets, and then sends them to the upper-layer system in the correct order.
Network System blocking will lead to the loss of data packets. The capacity of any network can be attributed to the transmission capacity or Routing Capability of the underlying physical layer.
NOTE: If network congestion occurs, some data packets may be lost. TCP resends the lost data packets, and the network congestion will become more and more serious. Therefore, if network congestion occurs, the retransmission speed of data packets will be reduced.

Although TCP emphasizes this mechanism, rfc793 does not use any rules to calculate the latency between data packets.

Iv. Temporary Analysis

Principle 4.1
The retransmission of data packets provides us with another way to analyze the remote host operating system. The fingerprint of the remote host operating system will be determined by analyzing the latency of the target host in each retransmission datagram.
This method has been defined in rfc793, but it leaves some room for use. Moreover, some implementations do not fully follow the current standards.
To let us know the role of this method, that is, the timeout mechanism, we must emphasize that the IP stack of the remote host must be in a non-standard state.
This situation can be easily imitated, simply give up the confirmation of the remote host SYN-ACK datagram can be achieved.
By measuring the delay between two adjacent datagram data records of retransmission, or observing some other information, such as TCP tag, serial number, and validation number, you can obtain some useful information about the remote host operating system.
If each operating system has its own characteristics, it is possible to establish a typical system identification database. Regardless of the host or network conditions, the operating system is the only factor that affects the test results. Therefore, it will be the same to detect the results of using the same operating system on different hosts (on the premise that the network conditions are relatively stable)
By comparing the target host fingerprint with the operating system fingerprint database, you may know the operating system running on the remote host.
The new operating system IDs will be easily recorded and will be associated with the operating system names.

DIY 4.2
This operating system fingerprint detection technology will use two parts of the probe instrument. A datagram filtering device, such as a personal firewall, and a datagram listening device.

Preparation of the probe instrument: A simple method to simulate network congestion is to install a personal firewall on the scanning host, and set some filtering rules to prohibit receiving all data from the target host. In this case, the datagram listener will receive all the data packets from the target host. Because of the existence of the firewall, the scan host will not send any confirmation information, so that a complete TCP connection cannot be established. After a certain period of time, the remote host will terminate any relationship with the scan host.

Test process: the test process consists of three different steps.
-Firewall settings
-Try to establish a standard connection with the target host
-The target host sends a confirmation datagram.

The implementation process is described in detail below:
-Select a host and confirm an opened port. For example, if the remote host is 192.168.0.10, its TCP port 80 is enabled (the web service enabled by the system );
-Configure the firewall to block all data packets from the target host (192.168.0.10.
-Listen to all data packets from Port tcp80 of the target host;
-Send a SYN datagram to the target host and try to establish a connection with it;
-Analyze the latency between all adjacent data packets from the target host.

In fact, what we measure is not the interval between sending continuous data packets to the target host, but the interval between consecutive data packets received from the target host. If these time intervals are almost fixed, we can boldly assume that they are equal.
The following figure shows the flow of data reported during the test (A indicates scanning the host, B indicates the target host to be tested, and a and B indicate no data transmission in a straight line without arrows ):

A B
│ -------- SYN ---------> │
│ SYN-ACK ------- │ --------------
│ --------------------- │ The interval is T1.
│ SYN-ACK ------- │ --------------
│ --------------------- │ The time interval is T2.
│ --------------------- │
│ SYN-ACK ------- │ --------------
│ --------------------- │
│ --------------------- │ The time interval is T3.
│ --------------------- │
│ --------------------- │
│ SYN-ACK ------- │ --------------

4.3 Static Analysis
Because the datagram may pass through some unstable networks, such as the Internet, it is very likely that the delay between the received continuous datagram (r_ I) and the delay when they are sent (s_ I) it is no longer the same.
If we receive two consecutive data reports from the target host, the measured latency is 3.01 seconds, but it is very likely that the latency between them is 3.0 seconds. The problem is that some systems use 3.2 seconds as the time delay for sending two consecutive data packets. Because the interval between 3.0 seconds and 3.2 seconds is too small, it is impossible to tell which system the received datagram comes from.
To avoid this problem, we can use the TCP time cut option to better obtain the information about when to send the datagram. Setting the time cut option in each datagram increases the accuracy of the test results. However, using the time cut option may increase the difficulty of our judgment, because the time cut option may return some inaccurate time data.

This method is based on Fingerprint acquisition and comparison with the existing fingerprint database. The general method of measuring the "distance" is to obtain the difference between the corresponding latencies:
Distance = Σ │ r_ I-s_ I │
Here, r_ I is the time delay related to the received I-th datagram, and s_ I is the time delay of the corresponding sequential datagram in the fingerprint database.
Therefore, the most likely operating system to be detected will be the operating system corresponding to the fingerprint database with the "shortest distance. However, the obtained "distance" does not take into account some important marks of the TCP datagram header (such as SYN, ack, RST, Fin ......) . These tags often detect the running status of the target host. Serial number and validation number, which can be used to differentiate different system implementation methods.

V. Ring execution and actual results

5.1 Database
For ease of development, ring uses the Standard C language and some library files in UNIX. For example, Dug Song's libdnet library, Mike D. Schiffman Libnet library, and Lawrence Berkeley National liboratory libpcap library.
The libdnet library is used to control the firewall. It provides an API to control some UNIX firewalls (such as ipchains, ipfilters, IPF ......)
Libpcap is usually used for network listening and datagram analysis.

5.2 running process
Ring needs some initialization parameters for remote host detection, including the IP address of the target host, an open TCP port on the target host, scanning the host's IP address, and the network interface used to listen for the datagram.
Then, the ring will perform the following steps:
-Select the source port;
-Use libdnet to establish a local Filtering Facility to block data packets from the target host;
-Use libpcap to listen for received data packets;
-Send a tcp syn data report to the target host using Libnet;
-Listen for the send-back datagram within the default/adjusted Delay
-Compare the received return datagram with the known Signature

5.3 actual results
When other tools cannot differentiate remote host systems, the ring still provides very accurate results. For example, a Win2k host and a freebsd host, even if they are hidden behind a normally configured firewall, the ring may also distinguish them.
The implementation of Win2k and FreeBSD is very similar because they share the same IP stack technology. If the two operating systems have only one open port, NMAP cannot distinguish the two.
If the implementation technology is very familiar, We will detect another reset datagram for further analysis, which should be sufficient for the ring to distinguish the operating system of the remote host.
The following is a comparison between Win2k and FreeBSD:
(In the following table, the data of each system is the time delay of retransmission datagram, in seconds. For example, the first line of data: 1 st indicates the first retransmission of the remote host, and 3 under Windows 2 k indicates that the first retransmission delay of Windows 2 K system is 3 seconds, another 3 in FreeBSD 4.4 also indicates that the first retransmission delay of FreeBSD 4.4 is 3 seconds .)
____________________________________________
│ Retries │ === windows 2 k ===│ = FreeBSD 4.4 = │
│ ===1st ===│ =========== 3 ===========│ ========= 3 =======│
│ ==2nd ===│ =========== 6 ===========│ ======= 6 =======│
│ == 3rd = │ = no more retries = │ ==== 12 ====== │
│ ===4th =================================│
----------------------------------------------------
│ = Reset = │ = No Reset sent = │ = reset after 30 sec. = │
----------------------------------------------------

Ring can also differentiate different versions of the same operating system:
-------------------------------------------------
│ Retries │ === linux2.2.14 ===│ === Linux 2.4 ==== │
│ ===1st ===│ =========3, 5 =========│ =========4, 26 ====== │
│ == 2nd = │ =========6, 5 =======│ ============= 6 ======== │
│ ===3rd ===│ =======12, 5 ===========│ =========12 =========│
│ ===4th ===│ =======24, 5 =========│ =========24 =========│
│ ===5th ===│ =======48,5 =========│ ===========48,2 ======== │
│ ===6th ===│ =======96, 5 =======│ = no more retries = │
│ ===7th ===│ =======120,5 =======│ =======================│
│ == 8th = │ = no more retries = │ =========================│
-------------------------------------------------
│ = Reset = │ === no reset =====│ === no reset ==== │
-------------------------------------------------

-------------------------------------------------
│ Retries │ === Windows 98 =====│ === windows 2 k ===│
│ ===1st ===│ =========== 3 ===========│ ============= 3 ======== = │
│ ==2nd ===│ =========== 6 ===========│ ============= 6 ====== = │
│ ===3rd ===│ ====== 12 ===========│ = no more retries = │
│ ==4th ==│ = no more retries = │ ===========================│
-------------------------------------------------
│ = Reset = │ ===== no reset =====│ ==== no reset ===│
-------------------------------------------------

The following are the differences between different devices:
---------------------------------------------------------
│ Retries │ Minolta printer │ Cisco router │ 3Com Switch │
│ ===1st ===│ =======4, 5 =======│ ========= 2 =========│ === = 3, 5 = │
│ ==2nd ===│ =======4, 5 =======│ ====== 3, 9 =====│ ==== 4 ==== │
│ == 3rd = │ ========= 9 ===========│ ======= 5, 9 ====== │ === = 4, 4 = │
│ ===4th ===│ =======18 =========│ no more retries │ =====4, 4 ==== │
│ ===5th ===│ =======36 =========│ =====================│ ==4,4 ==== │
│ ===6th ===│ =========72 =======│ =======================│ ==4,4 ==== │
│ ===7th ===│ ====== 144 =========│ =====================│ = 4, 4 = │
│ == 8th = │ ======= 285 =========│ =====================│ = 4, 4 = │
│ ===9th ===│ ====== 576 =========│ =====================│ = 4, 4 = │
│ ===10th = │ ========= 169 =======│ =======================│ 4, 4 = │
│ ===11th = │ ======= 169 =========│ =======================│ 4, 4 = │
│ ===12th = │ ======= 169 =========│ =======================│ 4, 4 = │
---------------------------------------------------------
│ = Reset = │ ==== reset =====│ === no reset =====│ = no reset = │
---------------------------------------------------------
Note: after a series of retransmission, some systems will send an rst datagram warning to scan the host to terminate the transmission, and then stop the retransmission datagram.

6. Discussion and Expansion

6.1 advantages
The biggest advantage of the ring technology is that it only needs an open port. If the target host is protected by the firewall, it is likely that only one port is enabled, and other ports are filtered out.
In the same firewall configuration, NMAP will not be so effective, because it is based on some closed ports, and these closed ports are often filtered out.
Moreover, the ring technique uses a standard TCP datagram, which will not adversely affect the target host.
On the other hand, this probe method takes more time than NMAP or xprobe. This is an inherent disadvantage of measuring the time delay of continuous datagram.
6.2 Protection
Is there any way to prevent the ring from detecting the operating system fingerprint? Because standard datagram is transmitted, it is impossible for the target host to split it from the common data transmission area.
The retransmission of data packets is obvious, but the loss and retransmission of data packets often occur on the network.
If an intrusion detection system suspends a connection to prevent excessive junk information on the network, it will reduce the TCP error rate and restore the network's circulation capacity.
In some operating systems, you can modify the data of some members of the TCP/IP stack to avoid ring detection. However, I do not recommend this method because it seriously threatens the stability of the TCP/IP stack.
Another possible method is to hide the host after the proxy, or use firewall technology to implement SYN forwarding or SYN gateway technology. SYN forwarding or SYN protection technology is specifically used to deal with SYN flood attacks.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
SYN forwarding principle:
(Note: In the following figures, the same host has the same name, and the straight line without arrows makes no sense. For example, three clients refer to the same client, and three firewalls refer to the same firewall)

Client ------ SYN ------> firewall -------------- target
Client <--- SYN-ACK ----- firewall -------------- target

If the firewall does not receive the waiting ack datagram within a certain period of time, the connection will be aborted:

Client <------ rst ------- firewall ---------------- target

Instead, the transmission continues:

Client ---------------- firewall ------- SYN ------> Target
Client ------ ack ------> firewall <--- SYN-ACK ----- target
Client ---------------- firewall ------- ack ------> Target

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
SYN gateway principle:

Client ------ SYN ------> firewall ------- SYN ------> Target
Client <---- SYN-ACK ---- firewall <----- SYN-ACK ---- target

If the firewall does not receive the waiting ack datagram within a certain period of time, it will interrupt the connection:

Client ---------------- firewall ------ rst ------> Target

On the contrary, the connection will continue:

Client ------ ack ------> firewall ------ ack ------> Target

6.3 future improvement
In the TCP transmission stream, another State also has the same function, trying to resend the data packets that are pretended to be discarded. This is the fin_wait_1 state. It can be used in combination with the previous detection technology, or in the test of the SYN protection system. This interesting technique can implement the SYN gateway protection mechanism to detect the system fingerprints of the real target host.
---------------------------------------------
│ Retries │ === linux2/4 =====│ = Windows 2 k ===│
│ == 1st = │ ====== 0, 8 =======│ =========== 3 ====== │
│ ==2nd ===│ ====== 1, 3 =======│ ========== 6 ====== │
│ == 3rd = │ ======= 2, 6 =======│ ======= 12 =====│
│ ===4th ===│ =======5, 2 =======│ ======= 24 =======│
│ ==5th ===│ =======10, 5 =========│ ====== 48 =====│
│ ===6th ===│ =====20, 8 =======│ no more retries │
│ ==7th ===│ =====41,6 =========│ =====================│
│ ==8th = │ no more retries │ ==================== │
---------------------------------------------
│ = Reset = │ === no reset ===│
---------------------------------------------

VII. Conclusion

The ring uses a new operating system detection technology built on conventional and non-hazardous TCP transmission. It identifies the operating system fingerprint of the remote host by analyzing the latency of the target host in each retransmission datagram. If it is combined with other operating system detection technologies, it should receive better results.

Reference:
1) <ring-full-paper> by intranode Research Team
2) <remote OS Detection via TCP/IP stack fingerprinting> by Fyodor
3) <ICMP based remote OS TCP/IP stack fingerprinting techniques> by Ofir Arkin & Fyodor yarochkin