How to build an entry-level Intrusion Detection System

Generally, when enterprises or organizations are preparing to enter this field, they often choose to start with network-based IDS, because there are a lot of open source code and materials on the Internet, which is easier to implement, in addition, network-based IDS have strong adaptability. With the development experience of simple network IDs, it is much easier to move towards host-based IDS, distributed IDs, and intelligent IDs. Here, I will take network-based IDS as an example to introduce typical IDs development ideas.

According to the cidf specification, the intrusion detection system is divided into four basic parts: data collection subsystem, data analysis subsystem, console subsystem, and database management subsystem, as shown in the figure below.

Generally, the Data Collection Subsystem (also called Detector) and data analysis subsystem are implemented on Linux or UNIX platforms; the console subsystem is implemented on Windows NT or 2000. The database management subsystem is combined with the console subsystem based on access or other more powerful databases. This is called the control management center. This article introduces the implementation of the data collection and analysis center and control management center on Linux and Windows NT platform as an example.

You can follow the steps below to build a basic intrusion detection system.

Step 1 get libpcap and tcpdump

Audit trails are the data source of IDs, and the data collection mechanism is the basis for implementing IDs. The data collection subsystem is located at the bottom layer of IDs. It mainly aims to obtain events from the network environment and provide events to other parts. Currently, libpcap and tcpdump are widely used to place NICs in the "hybrid" mode to capture all data streams on a certain network segment.

Libpcap is an essential tool for Unix or Linux to capture network data packets from the kernel. It is a system-independent API interface and provides a portable framework for underlying network monitoring, it can be used for network statistics collection, security monitoring, network debugging, and other applications.

Tcpdump is a tool used for network monitoring. It may be the most famous sniffer in Unix. Based on the Libpcap interface, tcpdump prints the packet header by applying a Boolean expression, perform the filter conversion, package acquisition, and package display functions. Tcpdump can help us describe the normal behavior of the system and ultimately identify abnormal behavior. Of course, it is only beneficial to collect data streams (network stream types, connections, etc.) on a certain network segment) information, as to whether the network activity is normal, it is the job done by programmers and administrators.

Libpcap and tcpdump are widely spread on the Internet. developers can download them from related websites.

Step 2: Build and configure the detector for data collection

1. you should select appropriate software and hardware devices based on your network conditions. If your network data traffic is small, you can install linux on a general PC, if the network traffic monitored is very large, you need to use a machine with high performance.

2. Open a log partition on the Linux server for storing collected data.

3. Create the libpcap library. Download from the Internet is usually a libpcap.tar. Z compressed package. Therefore, you should first decompress and decompress it, then execute the configuration script, create a makefile suitable for your system environment, and then use the make command to create the libpcap library. After libpcap is installed, a libpcap library, three include files, and a man page (User Manual) are generated ).

4. Create tcpdump. As with the Libpcap creation process, first extract and unpackage the compressed package to the same parent directory as libpcap, and then configure and install tcpdump.

If the configuration, creation, and installation operations are all normal, the system can collect network data streams here. For how to use libpcap and tcpdump, You need to refer to the relevant user manual.

Step 3: create a data analysis module

There are some open-source data analysis software packages on the Internet, which provides some convenience for us to build a data analysis module, but these "free lunch" generally have great limitations, to develop a truly powerful and practical IDs, developers usually need to design their own data analysis modules, which is often the focus of the entire IDs.

The data analysis module is equivalent to the IDS brain, which must be highly "intelligent" and "judgment ability ". Therefore, before designing this module, developers need to have a clear and in-depth study on various network protocols, system vulnerabilities, attack techniques, and suspicious behaviors, then, the corresponding security rule library and security policy are formulated, and the abuse detection model and exception detection model are established respectively, so that the machine can simulate its own analysis process and identify the attacks and abnormal behaviors of identified features, finally, an alarm message is generated based on the analysis results and sent to the control center.

The design of the data analysis module has a huge workload. Considering the increasing number of hackers, this process is doomed to be an endless process, update, upgrade, and improve constantly. Pay special attention to the following three issues:
① The design of Detection Models and Algorithms should be optimized to ensure system execution efficiency;
② The formulation of security rules should fully consider inclusiveness and scalability to improve system scalability;
③ Alarm messages must follow a specific standard format to enhance their sharing and interoperability capabilities. do not develop nonstandard message formats at will.

Step 4: Build the console subsystem

The console subsystem is responsible for reporting various network violations to the network administrator, and the Administrator takes actions (such as blocking and tracking) on some malicious behaviors ). Because Linux or UNIX platforms are far less popular than common Windows products in terms of supporting interface operations, to make IDs a general and easy-to-use system, I suggest implementing the console subsystem on Windows platforms.

The console has two main tasks:
① Manage the Data Collection and Analysis Center, and display alarm messages sent from the data collection and analysis center in a friendly and easy-to-query manner;
② Perform a series of response actions based on security policies to prevent illegal behaviors and ensure network security.

The console subsystem focuses on alarm information query, detector management, rule management, and user management.

1. alarm information query: the network administrator can use a single condition or compound condition to query the alarm information. When the alarm information is large and has a wide range of sources, the system must classify the alarm information according to the hazard level, this highlights the most important information required by the network administrator.

2. detector management: the console can manage multiple detectors (including start, stop, configuration, and running status) at a time, query the security status of each network segment, and formulate corresponding security rules for different situations.

3. Rule repository management function: provides users with a tool to flexibly configure security policies based on the specific conditions of different network segments, such as a custom tool that can be applied to multiple detectors and default security rules.

4. User Management: strictly defines user permissions, and provides functions such as password modification, user addition, user deletion, and user permission configuration to effectively protect the security of the system.

Step 5: Build a Database Management Subsystem

A good intrusion detection system should not only provide administrators with real-time and rich alarm information, but also detailed records of on-site data, so that some network events can be rebuilt in the future when evidence is required.

The front-end program of the database management subsystem is usually integrated with the console subsystem, using access or other databases to store alarm information and other data. The module has two data sources:
① Alarm information and other important information sent from the data analysis subsystem;
② After a conditional query, the Administrator processes the data obtained from the query result, such as the generated local file and format report.

Step 6: Joint debugging. A basic IDs has been set up.

After completing the preceding steps, the most basic framework of an IDS has been implemented. However, to make the IDS run smoothly, it is also necessary to ensure secure and smooth communication and interaction between all parts. This is the problem to be solved in the joint debugging work.

First, two-way communication is required between the data collection and analysis center and the control center. The control management center displays and sorts out the analysis results and other information sent by the Data Collection and Analysis Center. The data collection and analysis center receives configuration and management commands from the control management center. Ensure the communication security between the two. It is best to encrypt the communication data stream to prevent eavesdropping or tampering. At the same time, there are also a large number of interactive operations between the console subsystem and the database subsystem of the control management center, such as alarm information query and network event reconstruction.

After the joint debugging is passed, a basic IDS is set up. The next step is to constantly improve various functions, especially to improve the system's detection capability.