[Reprinted] streaminsight series-what it is

Link: http://blog.joycode.com/svs/archives/2009/12/24/115826.joy

SQL Server is a set of RDBMS. The functions of this product are not much said. In the real world, databases are often used as data storage and analysis platforms, however, some application scenarios are not suitable for database analysis and processing, especially for scenarios with high processing capacity and low latency. Currently, data in SQL Server is persisted through memory + disk. In other words, data in SQL will certainly be persistent, however, in the fields of manufacturing, monitoring, power supply, and finance, more data may not need to be persisted in real time, but must be monitored in real time, this is the function of the database and there are many restrictions.

Microsoft streaminsight provides a powerful platform for developing and deploying complex event processing (CEP) applications. CEP is a technology for high-throughput, low-latency processing of event streams. typical event stream sources include data from manufacturing applications, financial trading applications, web analytics, or operational analytics. the streaminsight stream processing architecture and the familiar. net-based development platform enable developers to quickly implement robust and highly efficient event processing applications.

Streaminsight is a framework

Many real-time monitoring event processing cannot use databases to implement these functions, because the data feature is not a transaction line, but an event. What we need at this time is event-driven processing.Program. Streaminsight is such a framework through which we can build robust and high-performance event processing programs, it can be used in analysis systems with high requirements for real-time data analysis, such as manufacturing, finance, power supply, and Internet.

For example, you can understand it very well. A car manufacturer assembles parts in several pipelines. In order to improve efficiency, many robots are used for welding and Assembly. In order to improve the product qualification rate, each robot's robotic arm has several sensors to monitor the solder joint position information. The sensor detects the trajectory and solder joint position of each motion of the manipulator and sends the information to the monitoring system. To improve the product quality, the monitoring program must observe the information sent by the sensor and match the information with the pre-designed indicators. This information processing delay is very high, and the throughput is very high, 10000 events/second, the latency is close to 0. If the event information is deviated from the preset indicator discovery and the deviation is higher than the preset range, the system should trigger an alarm. If the design of such a system is implemented using a relational database, the performance will not be satisfied. Streaminsight can solve this problem. event data is sent to the streaminsight component in queue and stored in the memory until the business rules are met and then retrieved from the output.

Streaminsight is used to process events.

There are 2 types of events in streaminsight

• Insert

Insert events are used to add events and payload to the event stream.

Header
Payload

Event kind ::= insert

Starttime: =Datetime

Endtime: =Datetime

Field 1... FieldNAs CLR types

• CTI

CTI events are special events used to indicate the completion of events.

Header

Event kind ::= CTI

Starttime: =Datetime

That is to say, after the event is inserted into the queue, a CTI must be added to indicate that the event is over.

In streaminsight, events have three models:

• Point
  

  This kind of event features Payload Based on time points, such as 3:00:00 temperature 24

• Interval
  

  The feature of this event is to describe the payload for a period of time, for example, 3:00:00 2009-12-24 3:00:10 traffic flow 100

• Edge
  

  The feature of this event is that no end time is specified in the record, that is, the type of the event in the record, including start and end

We can input and output the three types of events through streaminsight.

Streaminsight core is event-based payload Query

The core function of streaminsight is to process events based on rules, mainly to summarize, calculate, filter, and project events based on fields in payload, these functions are based on the LINQ query of the event queue in the memory. Of course, these queries are not necessarily hard code and can be dynamically adjusted by encoding.

Streaminsight is not a memory database

Some people think that streaminsight is a memory database. I don't think this is correct. Because streaminsight is only used to process events and such events must be event-based. Although we do not need to store event fields in payload, the system will add time fields to assist the system in processing. In-memory databases require more functions, such as starting data loading and persistence, and implementing crud functions, but streaminsight does not. However, in some business scenarios, you can use streaminsight to implement the memory database function.

Comment: author Sun. wei has a profound skill. This streaminsight series blog first introduces streaminsight and integrates his own understanding. I think this blog will surely inspire readers who will learn streaminsight.