Evaluation of IDS intrusion detection system

With the wide application of intrusion detection system, the requirement of testing and evaluating intrusion detection system is more and more urgent. Developers want to test and evaluate the deficiencies in the product, users want to test and evaluate to help themselves choose the right intrusion detection products. Based on the current research, this paper introduces the criteria, indexes, method steps, data sources, environment configuration, test evaluation status and some problems in the intrusion detection system test evaluation.

1 Introduction

With the gradual improvement of people's safety consciousness, intrusion detection system (IDS) has become more and more widely used, and various IDs are becoming more and more. So is IDs able to detect intrusion behavior? Does IDs meet the developer's design goals? What IDs are the performance IDs that users need? To answer these questions, test and evaluate IDs.

As with other products, when IDs is developed and applied to a certain extent, the requirements for testing and evaluating IDs are put on the agenda. All parties want to have convenient tools, a reasonable way to science of IDs. Test and evaluate fairly and credibly. For the development of IDs and developers, the various IDs of the regular assessment, can be timely understanding of the status of technology development and system deficiencies, so focus on those key technical issues, reduce system deficiencies, improve the performance of the system, but for the users of IDs, Because of their increasing reliance on IDs, they also want to use the assessment to choose the products that suit their needs and avoid misleading the IDs product hype. Users of IDs are particularly eager to test assessments because most users may not be very knowledgeable about IDs itself, and they want expert evaluation results to be the basis for their choice of IDs.

Overall, testing and evaluating IDs has the following effects:

• Helps to better characterize the IDs. Through the test evaluation, we can better understand the processing method, resource and environment of IDs, establish the benchmark of comparing IDs, and comprehend the relationship between each detection method.

• Evaluate performance of IDs and determine the performance level of IDs and their impact on the operating environment.

• Use of testing and evaluation results to make predictions that infer trends in IDS development, estimate risk, and develop achievable IDs quality objectives (e.g., reliability, availability, speed, accuracy), cost, and development progress.

• Improve IDs based on test and evaluation results. That is to find the problems in the system and to improve the system to improve the performance indicators.

Firstly, this paper introduces the standard of testing and evaluating IDs performance, then introduces the method steps of test evaluation, introduces the specific indexes of test evaluation, the data sources needed, the configuration and framework of test evaluation environment, and finally introduces the present situation of test evaluation and some problems in it.

2 testing criteria for assessing IDs performance

According to the research of Porras, this paper gives three factors to evaluate the performance of IDs:

• Accuracy (accuracy): Refers to IDs from a variety of actions to correctly identify the ability to invade, when an IDs detection is inaccurate, it is possible to the system of legitimate activities as an intrusion behavior and identified as abnormal (false alarm phenomenon).

• Processing performance (performance): The speed at which an IDs processes data source data. Obviously, when the performance of IDs is poor, it is impossible to realize real-time IDs, and it may become the bottleneck of the whole system, which will seriously affect the performance of the whole system.

• Completeness (completeness): refers to the ability of IDs to detect all attack behaviors. If there is an attack behavior that cannot be detected by IDs, then the JDS does not have detection completeness. In other words, it treats the intrusion activity of the system as normal behavior (omission phenomenon). It is difficult to evaluate the completeness of IDs, because in general, the types of attacks and the means of attack are rapidly changing, and we find it difficult to get all the knowledge about the attack behavior.

On this basis, Debar has added two performance evaluation measures:

• Fault tolerance (Fault tolerance): Because IDs is an important means of detecting intrusions/So it has become the preferred target for many intruders to attack. IDS itself must be able to withstand attacks on itself, particularly denial of service (Denial-of-service) attacks. Since most IDs are run on highly vulnerable operating systems and hardware platforms, this makes system fault-tolerance particularly important and must be taken into account when testing IDs.

• Timeliness (timeliness): timeliness requires the IDs to analyze the data as quickly as possible and spread the results so that the system security Manager can react before an intrusion attack has done more damage and prevent further sabotage by the intruder, compared with the above performance factors, The requirement of timeliness is higher. It not only requires the processing speed of IDs to be as fast as possible, but also requires less time to propagate and respond to test results.

3 method steps for IDs test evaluation

We have discussed the performance metrics of IDs test evaluations, and the specific tests are focused around these metrics. Most of the test procedures follow the following basic test steps:

• Create and select some test tools or test scripts. These scripts and tools are used primarily to generate simulated normal behavior and intrusion, which is the actual environment that simulates the operation of IDs.

• Determine the conditions required by the computing environment, such as the level of background computer activity.

• Configure run IDs.

• Run test tools or test scripts.

• Analyze the detection results of IDs.

Nicholas J.puketza of the University of California, USA, and others divide the tests into three categories, which correspond to the previous performance indicators, i.e. intrusion detection tests (or IDs validity tests). Resource consumption test, strength test. Intrusion detection test measures the ability of IDs to distinguish between normal behavior and intrusion, and the main measures are detecting rate and false alarm rate. The resource consumption test (Resource Usage Tests) measures the status of the system resources taken up by IDs, and the main factors considered are hard disk footprint, memory consumption, and so on. The strength test mainly detects whether IDs is affected by the test effect under the condition of strong load, mainly including the detection effect under the condition of heavy load and high density data flow.

4 Test performance metrics for IDs

When we analyze the performance of IDs, we mainly consider the effectiveness, efficiency and usability of the detection system. Effectiveness studies the detection accuracy of the detection mechanism and the reliability of the system test results, it is the premise and purpose of development design and application of IDs, is the main index to test and evaluate IDs, and efficiency is considered from the point of view of processing data and economy of the detection mechanism, which is to emphasize the improvement of the performance price ratio of the detection mechanism. Usability mainly includes the scalability of the system, the usability of the user interface, the convenience of deployment configuration, and so on. Effectiveness is the premise and purpose of developing the design and application of IDs, and therefore the main indicator for testing IDs, but efficiency and availability also play an important role in the performance of IDs. Efficiency and usability permeate all aspects of system design. This section analyzes the performance metrics for testing the IDs in terms of the effectiveness, efficiency, and usability of the test.

4.1 Detection rate, false alarm rate and detection credibility

Detection rate refers to the probability that the detection system can correctly alarm the monitored system when it is attacked by an intrusion. False alarm rate refers to the probability of false alarm in the detection system. Detection credibility is the credibility of the detection system test results, which is the most important measure to evaluate IDs.

The realization of the actual IDs is always hovering between the detection rate and false alarm rate, the detection rate is high, the false alarm rate will be increased; the same false alarm rate, the detection rate will be reduced. In general, IDs products take a tradeoff between the two and can be adjusted to suit different network environments. The Lincoln Laboratory in the United States describes the performance of IDs using the receiver feature (roc,receiver operating characteristic) curve. This curve accurately depicts the relationship between the detection rate and false alarm rate of IDs. ROC is widely used to input the evaluation of uncertain systems. According to the false alarm rate and detection rate under the different conditions of IDs (such as the threshold value of the anomaly detection system, such as the alarm threshold, etc.), the false alarm rate and the detection rate are used as horizontal coordinates and ordinate, and the ROC curve corresponding to the IDs can be made. The ROC curve has a corresponding relationship with the detection threshold of IDs.

In the process of testing and evaluating IDs, in addition to the detection rate and false alarm rate of IDs, some factors related to these two indexes are considered separately, such as the number of intrusion features, the ability of IP fragment reorganization and the ability of TCP flow reorganization. Obviously, the more intrusion features can be detected, the higher the detection rate will be. In addition, some specially designed groupings are often sent for attackers to increase the difficulty of detection and even bypass IDs detection. In order to improve IDs detection rate and reduce the false alarm rate of IDs, IDs often need to take some appropriate measures, such as IP fragmentation capability, TCP stream reorganization. Because analyzing individual data groupings leads to many false positives and false negatives, the reorganization of IP fragments can improve the accuracy of detection. The IP Fragmentation evaluation standard has three performance parameters: the maximum number of IP fragments that can be reorganized, the number of IP groupings that can be reorganized at the same time, the length of the maximum IP data packet that can be reorganized, and the TCP stream reorganization is to analyze the complete network dialogue, which is the basis of the analysis of the application layer of the network IDs. such as checking the contents of the message. Attachments, check FTP transmitted data, prohibit access to harmful web sites, and Judge illegal HTTP requests. Both of these capabilities directly affect the detection credibility of IDs.

4.2 IDs itself against attack capability

As with other systems, IDS itself often have security vulnerabilities. If the attack on IDs succeeds, it directly causes the alarm to malfunction, and the behavior of the intruder will not be recorded. Therefore IDs must first ensure its own security. The robustness of IDs itself is the reliability of IDs, which is used to measure the resistance of IDs to attacks that are specifically designed to target IDs directly. It is mainly embodied in two aspects: first, the program itself in a variety of network environment can work properly; the second is that the communication between each module of the program can not be destroyed, not counterfeit. In addition, special consideration should be given to the ability to withstand denial of service attacks. If the IDs itself does not function properly, it loses its protective meaning. And if the communication between the modules of the system is destroyed, the system of alarm and such detection results are questionable, there should be a good communication mechanism to ensure that the communication between the modules of the security and can be quickly recovered in the event of a problem.

4.3 Other performance indicators

Delay time. Detection latency refers to the delay between the time the attack occurred and the intrusion detected by IDs. The length of delay time is directly related to the extent of intrusion attack damage.

Resource occupancy. That is, the system's demand for resources when it achieves some kind of detection effectiveness. In general, under the premise of the same detection efficiency, the lower the requirement for resources, the better the performance of IDs and the stronger the ability of detecting intrusion.

Load capacity. IDS has its design load capacity, in the case of exceeding the load capacity, performance will appear to varying degrees of decline. For example, under normal circumstances IDs can detect an attack but may not detect it under heavy load. Examining the load capacity of the test system is to observe the influence of the different size network traffic, the different intensity CPU memory and other system resources on the key index of IDs (such as detection rate, false alarm rate).

Journal, good reporting, report, and responsiveness. Log capability is the ability to detect the ability of the system to save logs, and to select the contents of the log according to specific requirements. Alarm capability refers to the ability to send alarm signals to special parts and personnel after intrusion detection, and to attach information to the alarm. Reporting capability refers to the ability to generate intrusion behavior reports, provide query reports, create and save reports. Responsiveness refers to the ability to further process after detection of an intrusion, which includes blocking intrusion, tracking intruders, recording intrusion evidence, and so on.

The availability of the system. Mainly refers to the system installation, configuration, management, ease of use, system interface friendliness, attack rule base maintenance, such as the ease of the level.

In short, IDS is a complex system, and testing and evaluating IDs is related not only to IDs itself, but also to the environment in which IDs are applied. The testing process involves operating environment, network environment, tools, software, hardware and so on. We must consider the effect of intrusion detection, but also consider the application of the system after its impact on the actual system, and sometimes to compromise consideration of these two factors.