In-depth introduction to event stream processing nesper (1)

For real-time information analysis and processing, it is often necessary to develop the corresponding functions of the customer application. In general, these functions need to provide the following processing procedures, analyze the acquired data, filter the data, extract useful information, and then display it in a specific form. Because of the high concurrency and high throughput requirements of specific real-time information, this requires the customer's applications to have high scalability and response capabilities, but in the field of data processing. Nesper is A. Net open-source event stream processing solution for such problems. It aims to simplify the development of customer applications with this requirement.

1.1.cep and event stream

The purpose of nesper is to provide applications with Event Analysis and Response requirements. Typical application requirements are as follows:

• Business Process Management and automation. (Process monitoring and exception reporting are provided)

• Finance (algorithm trading, fraud detection, risk management)

• Network and application monitoring (intrusion detection and SLA (Service Level Agreement) Monitoring)

• Sensor network applications (RFID reading, production line scheduling and control, air traffic)

These applications have the need to process real-time or similar real-time events, sometimes known as complex event processing and event stream analysis. Its business events have key factors such as throughput, latency, and business complexity.

• High Throughput-processing a large number of messages (Applications between 1000-kb of messages per second)

• Low latency-application response in real time (from milliseconds to seconds)

• Complex computing-the application detects related events based on rules, filters events, and adds events that meet the conditions to time-based event streams or window-based event streams, this operation is triggered when the event leaves.

The nesper event engine is used to construct such application requirements and expand complex event processing.

1.2.cep and relational databases

Relational databases and structured query languages aim to meet the data requirements of application systems that are relatively static and have complex query requirements. Therefore, most of these databases (except memory-based databases) store data on hard disks. Therefore, the optimization usually lies in the optimization of hard disk access.

Applications often use queries to obtain data in the database. If the application needs to query data 10 times per second, frequent data queries are generated. This method is not very good for hundreds of massive requests per second.

Database Triggers can respond to update operations accordingly. However, database Triggers are slow and cannot easily perform logical detection and response to complex conditions.

Compared with traditional relational databases, memory-based databases in terms of query performance may be more suitable for complex event processing applications. Then, the memory database is not optimized for such query performance requirements.

1.3. nesper Engine

The working principle of nesper is similar to that of database inversion. It is different from the traditional method of storing and querying data first. The nesper engine sets query conditions in advance and then allows real-time data to pass these query conditions. The engine captures data that meets the conditions, which is continuous.

It provides two methods to process these events, Event Mode and event flow query.

Nesper provides expression-based event matching rules. The pattern matching engine is an implementation of a state machine. This event is handled by matching and processing events that arrive or exit. It is based on the event stream of the time window.

Similarly, nesper provides another method. This window-based event stream provides aggregation, connection, and analysis functions to handle these complex events. Query this event window through EPL (a SQL-like syntax defined by nepser. Obtain the event that meets the conditions. EPL is similar to SQL, but it is different from SQL. Query Based on views rather than tables in EPL. The view provides many operations to obtain the corresponding data from the event stream.

Nesper provides corresponding APIs and supports the above two methods.

1.4 required third-party packages

The running environment of nesper requires the following third-party packages

• Anlr EPL syntax. The generation and interpretation tools that require access to: http://www.antlr.org during Environment compilation.

• Cglib is a dynamic generation tool based on open source bytecode licensed by. The Apache 2.0.

• Apache log4net open-source logging tool.

Third-party packages required by other compiling and testing environments of nesper

• Nunit is an excellent unit test Framework Based on. NET.

• MySQL/SQL server: the SQL test suite of nesper.

Related Articles:

Event stream processing framework nesper for. net