[Compilation] streaminsight Introduction

Link: http://blogs.msdn.com/ B /masimms/archive/2010/07/23/secundus-posting-of-sorts.aspx

Author: Mark Simms, Senior Program Manager of Microsoft SQL customer consulting team.

(Because there are outdated and irrelevant original texts, I have made some deletions and rewriting, and fixed the title as "Compilation" instead of "Translation ", for the differences between the two, see here .)

 

I haven't published a new article here for a while (well, frankly speakingLongTime is not published ). Since, my role at Microsoft has undergone some subtle changes. I will sit in the SQL customer advisory team to help customers and partners build large-scale event-driven applications using streaminsight technology.

As streaminsight was released as a subset of SQL Server 2008 R2, I finally spent enough time talking about the best practices we learned in this product after several amazing projects, we also hope to share some valuable information to help you build your own streaminsight application.

Okay. Let's take a look at streaminsight!

What is streaminsight?

Anyone familiar with me or who has heard of my speech knows that I love analogy (although sometimes it is bad), and it is no exception here. Note: The examples here are all set by humans. We will talk about some real scenarios later in this article.

If we want to answer this question: "How many red cars are parked in the parking lot behind your office ?", The physical and relational database methods can be used to answer the following question:

Physical Mode	Relational Database
Take a notebook and draw a pen to the parking lot; Walk through the parking lot to view every car; If the car is a passenger car and it is red, mark it on the paper; Count the number of all marks and go back to the office.	Connect to the target database; Execute SQL commands, suchSelect count (*) from parkinglot where type = 'auto' and color = 'red'.

Very straightforward practice! We can continue to think about how to optimize this query. For example, we can stop all the cars on the parking lot side first, and then arrange the parking according to the color (in relational database, we can consider adding an index ). We can see that we already have a good relational database platform for such problems. However, there is another kind of problem that the relational database engine cannot handle well.

Suppose the question is: "How many cars have passed your office in the last 20 minutes ?". Similarly, if we use physical and purely relational databases to answer questions, the result will be:

Physical Mode	Relational Database
Drag the vehicle on the highway to a waiting area; Hold each vehicle for at least 20 minutes and record who is allowed to pass; Count the number of vehicles each time a car enters or leaves.	Put records of incoming vehicles into the database; Continuously execute the following SQL commands, suchSelect count (*) from parkinglot where type = 'auto' and color = 'red'; Remove a vehicle (record) from the database periodically using a separate job.

It can be seen that the above two practices are not the best way to answer this question. Streaminsight fills this gap by providingStream ProcessingThe platform can effectively process large-scale streaming data. The same problem is solved using streaminsight:

Physical Mode	Streaminsight
Standing next to the road, count the number of cars that pass.	Use the following LINQ statement to set a continuously running query; Use the streaminsight engine to connect the query to a constant traffic event.

var x = from e in cars         where e.Type == "AUTO" && e.Color == "RED"        select e; var count = from e in x.HoppingWindow(                TimeSpan.FromMinutes(2),                TimeSpan.FromSeconds(30),                HoppingWindowOutputPolicy.ClipToWindowEnd)            select new             {                 count = e.Count()            };

The preceding LINQ statement is a streaminsight query. The procedure is as follows:

• View the flow data of the vehicles to be observed (CARS flow );
• Filter out any events that are not auto or red;
• Define the window time to view the last 2 minute events and move the window forward at an interval of 30 seconds;
• In each window, count the total number of events (that is, the number of red cars in the last two minutes) and report the total number.

Yes, and so on. Do you want to know how LINQ solves the data problem in transmission magic? Haha, this is the magic of streaminsight. It provides the following platforms:

• MemoryStream processing engine;
• Connect the stream data source (adapter) for continuous query;
• Inquire about temporary data and associated data (windows within a certain time period and associations within a certain time period), including dynamic links and combined queries;
• Extends queries, operators, aggregates, and data connectivity.

What is the use of streaminsight?

Streaminsight is designed to help users develop powerful large-scale event processing applications. This also determines that it can be applied to many different industrial fields, such:

Manufacturing	Process Analysis of sensor in workshop floor; The device controller responds to exceptions and trends;
Financial services	Customization of stocks and news; Algorithm transaction;
Operation Analysis	Traffic data and behavior analysis; User Experience Quality Analysis;
Public utilities	Smart grid Analysis for thousands of electric meters; Identify trends, usage modes, and potential error statuses;

Basically, streaminsight appliesLargeStreaming DataLow latencyThe application that gets the processing result.

It sounds good. How do I get started?

To get started with streaminsight, you need to be familiar with C # And LINQ and have Visual Studio (2008 or 2010 ). Once these are ready, you can start:

1. Download and install the latest version of streaminsight 1.2 from the Microsoft download center;
   1. 64-bit installation files and 64-bit Client installation files
   2. 32-bit installation files and 32-bit Client installation files

   Note: The Microsoft streaminsight client package only installs a subset of Microsoft streaminsight. It only allows users to write client-related functions: connect to the existing streaminsight server, manage queries, and use debugging tools. It does not support built-in servers or creating streaminsight services. In addition, you do not need the product key to install the Microsoft streaminsight client.

2. Download the reference sample from codeplex to the local hard disk;
   1. Because the files downloaded by codeplex (most of the files downloaded online) are untrusted by default, You need to first "unlock" and then decompress the folder;
   2. Right-click the streaminsightproductteamsamples compressed package and click "properties". You will see:
      
   3. Click "unlock ".
3. Open Applications \ trafficjoinquery. sln solution from Visual Studio;
4. The trafficjoinqueryapplication uses the simulated traffic data (the two. CSV files included in the solution) and executes the query listed in the createquerytemplate function (in the program. CS file;
5. Press F5 to debug the application. The application executes queries on two simulated data streams. The running results are similar to the following:
   

   Creating CEP Server Creating CEP Application Registering LINQ query Template Registering adapter factories Registering bound Query Start Query Insert 6/25/2009 12:00:00 AM + 1 1001 18 Insert 6/25/2009 12:00:00 AM + 1 1003 19 -- Snip -- Nsert 6/25/2009 12:18:20 AM + 2 1004 21 Insert 6/25/2009 12:18:20 AM + 2 1005 19 Insert 6/25/2009 12:18:20 AM + 2 1006 21 Insert 6/25/2009 12:18:20 AM + 3 1007 22 CTI 12/31/9999 11:59:59 + Diagnostic view for 'cep:/Server/eventmanager ': Alleventscount: 18 Alleventsmemory: 417792 Diagnostic view for 'cep:/Server/planmanager ': -- Snip -- Querytotalincomingeventcount: 497 Querytotalconsumedeventcount: 497 Querytotalproducedeventcount: 192 Querytotaloutgoingeventcount: 192 -- Snip --

   The preceding process shows the streaminsight Engine Startup, activation query, output results, and final export of diagnostic information.

What learning materials are there?

Next, I will post a series of blog posts to discuss how to build streaminsight applications, including adapter development, query writing, hosting streaminsight, and connecting to other technical platforms (such as appfabric cache. Now I will give some very good reference links for your reference:

• Streaminsight product blog
• Streaminsight Forum
• Streaminsight document of msdn
• Streaminsight query Guide