[Reprinted] streaminsight series-adapter

Link: http://blog.joycode.com/svs/archives/2009/12/25/115829.joy

It is really difficult to stick to post every day, unless it is the kind of Otaku or suffering from strong computer dependence. Writing continuously is really tiring.

Today, we will talk about the adapter in streaminsight. This is a big topic, but it is very important because the data input and output depend on it. Developing an adapter is not very difficult, but you need to first understand the working mechanism of the adapter. The adapter works in the state machine mode, and our code mainly implements switching between different States and how to obtain information from the data source and construct event data.

After the adpater is created, it calls the aapter start method through the query startup, and then the adapterstate enters the running state. In this case, we need to create and enqueue the event through code, enqueue returns a result indicating the status of the queue. If the status is suppended, streaminsight calls the resume () method to change the status of the adapter to running. The inputadapter and outputadapter statuses are shown in this figure.

The stop of the adapter is divided into two states: stopping and stopped. We should pay attention to the adapterstate when processing and Generating Events. If it is stopping, we need to process the data in the buffer zone.

The adapter must be instantiated in factory mode. First, we need to implement iinputadapterfactory \ itypeinputadapterfactory or the corresponding outputadapterfactory, which appears in pairs. These two interfaces are used to create weak and strong types of adapterfactory. I personally think the weak type is more flexible, the strong type has better performance, and can be compiled and checked. The specific usage needs to be combined with the needs. The following is my adpaterfactory code. Here, I only process the event of the point type and return the adapter networkpointinput.

Public struct networkinputconfig {// network listening Address Public String interfaceaddress {Get; set;} // CTI public int ctifrequency {Get; set ;} // enter the data field name and sequence public list <string> inputfieldorders {Get; set;} public class networkinputfactory: iinputadapterfactory <networkinputconfig> {public inputadapterbase create (networkinputconfig configinfo, eventshape shape, cepeventtype) {inputadapterbase adapter = default (inputadapterbase); If (eventshape = eventshape. point) {adapter = new networkpointinput (configinfo, cepeventtype);} else {Throw new argumentexception ("the adapter cannot instance adapter with event shape {0}", eventshape. tostring ();} return adapter;} public void dispose (){}}

For the adapter, We need to inherit the pointinputadapter class and reload the resume () and start () methods. Of course, we also need to write the constructor to accept the config and cepeventtype passed in adapterfactory.

Public networkpointinput (networkinputconfig configinfo, cepeventtype) {_ bindtimeeventtype = cepeventtype; _ ctifrequency = configinfo. ctifrequency; foreach (pcapdevice device in pcap. getalldevices () {foreach (pcapaddress Address in device. addresses) {If (address. ADDR. IPaddress. tostring () = configinfo. interfaceaddress) {_ DEVICE = device; break;} If (_ device! = NULL) break;} If (_ DEVICE = NULL) throw new argumentexception ("interface address is not found"); _ inputordinaltocepordinal = new dictionary <int, int> (); // The number of configuration file fields and the number of cepeventtype fields are inconsistent if (configinfo. inputfieldorders! = NULL & configinfo. inputfieldorders. Count! = Cepeventtype. fields. count) {Throw new argumentexception (string. format (cultureinfo. invariantculture, "the configuration element inputfieldorders shold have {0} elements, but it only has {1} elements", cepeventtype. fields. count, configinfo. inputfieldorders. count);} cepeventtypefield enginefield; For (INT I = 0; I <cepeventtype. fields. count; I ++) {If (configinfo. inputfieldorders! = NULL) {If (! Cepeventtype. fields. trygetvalue (configinfo. inputfieldorders [I], out enginefield) {Throw new argumentexception (0; string. format (cultureinfo. invariantculture, "event type {0} doesn't have an input field named '{1}'", cepeventtype. shortname, configinfo. inputfieldorders [I]);} _ inputordinaltocepordinal. add (I, enginefield. ordinal);} else {// use default mapping _ inputordinaltocepordinal. add (I, I );} If (! _ Device. Opened) _ device. Open (true, 1000); _ device. setfilter ("IP and TCP ");}

My program references the sharppcap class library and installs Winpcap to listen to network data.

The constructor is still very simple. It basically creates some internal objects to be used from the configuration file.

public override void Resume() {     ProduceEvents(); }public override void Start() {     ProduceEvents(); }

This section is mainly about produceevents ()

Private void produceevents () {pointevent currevent = default (pointevent); enqueueoperationresult result = enqueueoperationresult. Full; packet = NULL; while (true) {// determines the device status if (! _ Device. opened) {_ device. open (true, 1000); _ device. setfilter ("IP and TCP");} // obtain the network packet cyclically while (packet = _ device. getnextpacket ())! = NULL) {# region adapter stopping if (adapterstate. Stopping = adapterstate) {If (_ pendingevent! = NULL) {currevent = _ pendingevent; _ pendingevent = NULL;} preparetostop (currevent); stopped (); return;} If (_ pendingevent! = NULL) {currevent = _ pendingevent; _ pendingevent = NULL;} else if (_ pendingctitime! = NULL) {} else {If (packet is tcppacket) {currevent = createeventfrompacket (packet); _ pendingevent = NULL;} If (_ pendingctitime! = NULL) {result = enqueuectievent (_ pendingctitime. value); If (enqueueoperationresult. full = Result) {preparetoresume (_ pendingctitime. value); ready (); return;} else {_ pendingctitime = NULL;} else {datetimeoffset curreventtime = currevent. starttime; Result = enqueue (ref currevent); If (enqueueoperationresult. full = Result) {preparetoresume (currevent); ready (); return;} else {_ eventsenque UED ++; _ pendingevent = NULL; If (zero = (_ eventsenqueued % _ ctifrequency) {datetimeoffset currctitime = curreventtime + timespan. fromticks (numberofticks); Result = enqueuectievent (curreventtime); If (enqueueoperationresult. full = Result) {preparetoresume (curreventtime); ready (); return;} else {_ pendingctitime = NULL ;}}}} private pointevent createeventfrompacket (packet) {poi Ntevent EVT = createinsertevent (); EVT. starttime = datetime. now; tcppacket TCP = (tcppacket) packet; string content = string. empty; If (TCP. tcpdata. length> 0) {content = system. text. asciiencoding. ASCII. getstring (TCP. tcpdata); If (content. length> 256) content = content. substring (0,256);} string [] DATA = new string [] {// Len packet. pcapheader. capturelength. tostring (), TCP. sourceaddress. tostring (), TCP. destinationaddress. tostring (), TCP. sourceport. tostring (), TCP. destinationport. tostring (), content}; // populate the payload fields for (INT ordinal = 0; ordinal <_ bindtimeeventtype. fieldsbyordinal. count; ordinal ++) {try {int cepordinal = _ inputordinaltocepordinal [ordinal]; cepeventtypefield evtfield = _ bindtimeeventtype. fieldsbyordinal [cepordinal]; object value = convert. changetype (Data [Ordinal], evtfield. type. clrtype); EVT. setfield (cepordinal, value);} catch (adapterexception e) {console. writeline (E. message + "\ n" + E. stacktrace) ;}} return EVT;} private void preparetoresume (datetimeoffset currctitime) {_ pendingctitime = currctitime;} private void preparetostop (pointevent currevent) {If (null! = Currevent) {releaseevent (ref currevent);} If (_ device. Opened) _ device. Close ();} private void preparetoresume (pointevent currevent) {_ pendingevent = currevent ;}

These are the core parts. Basically all the adapters are similar. The difference is how to create an event. Here I have a method createeventfrompacket () to create an event. However, you must note that the maximum number of fields in event payload cannot be changed. By default, each field is up to 512 bytes. Currently, I do not know how to change the default length limit of payload.

@ Starbucks_guangzhou

Comments: The code for this input Adapter should be written in imitation of the csvinputadapter. Of course, there are a few explanations. Here we recommend that you read the streaminsight shouzha series from the SQL CRD msdn blog.