Quickfix/j Source Code Research (i)
Liyayawodeai@163.com
(0) quickfix/j Introduction
fix is the abbreviation for financial information exchange. Fix is a standard message protocol designed specifically for real-time electronic securities trading. The fix agreement is owned and maintained by Fix Protocol, LTD (FPL). The Fix protocol URL is http://www.fixprotocol.org
Quickfix/j is an open-source software that implements all the versions of the fix protocol and its functionality, and 100% uses Java implementations.
Quickfix/j's web site is http://www.quickfixj.org
Quickfix/j source code can be downloaded from the Http://sourceforge.net/projects/quickfixj/files/QuickFIX_J, you can also go to Quickfix/j's official website, into the download page download source codes.
So what can you do with quickfix/j? The brothers who pay attention to the stock must have paid attention to the term levelii, he is the abbreviation of the new market of Chinese stock trading. Simply put, you can change the quickfix/j code, to accept the Shenzhen Securities trading levelii Market data. Of course, accepting quotes is not free and requires a lot of business formalities, but this article only discusses the design and implementation of QUICKFIX/J open source code, and focuses on the QUICKFIX/J client implementation. The server side stays in later article introduction. On the Shanghai Stock exchange of the LEVELII data format and acceptance, with Shenzhen, there are many differences, also stay in the future discussion.
First of all, there are two main QUICKFIXJ code functions, part of the Fix protocol data analysis, and the other part of the client and server-side to establish a connection and maintain a reply, transmission data. The first part will mainly introduce the realization of quickfix/j transmission part.
(i) Quickfix/j transmission function Section
The QUICKFIX/J connection management and transmission function is based on the Mina framework. What Mina is. Mina is an Apache network application framework that can help you easily develop high-performance, highly scalable network programs. It uses NIO to provide an abstract, event-driven, asynchronous processing API on top of transport protocols such as TCP/IP,UDP/IP. Mina's web site is http://mina.apache.org.
A the functional description of the primary class used by the quickfix/j client (V1.5.0)
1. Quickfix. Initiator: Defines a number of time intervals for obtaining communication protocols, hosts, ports, and reconnection from the configuration.
KEY, just a key, nothing else.
2. Quickfix.mina.SessionConnector: Defines helper methods that provide common functions for initiator and acceptor, such as getting a session, creating a session, dynamically adding/deleting sessions To determine whether they have landed. In Sessionconnector, you define the
Sessiontimertask, the main purpose of this timer is to routinely check and update the status of the session, and find problems in a timely manner.
SessiontimertaskAfter discovering the login status error can automatically reset the login, update session timestamp, send heartbeat message. Check the specific logic in Session.next (), details please refer to 5.
3. Quickfix.mina.initiator.AbstractSocketInitiator: Socketinitiator is the base abstract base class, inheriting Sessionconnector and initiator. Two default implementations are provided in Quickfix/j, Socketinitiator and Threadedsocketinitiator respectively. Both implementations have the same functionality, and the difference is only that the thread's policy is different when processing messages, specifically refer to 7 and 8. The features provided by the abstract class Abstractsocketinitiator are:
A) traverse the configuration file to obtain the configuration of all [session] sections and create the corresponding fixsession (if no connectiontype is specified in [session] or the ConnectionType is explicitly specified as initiator, The Fixsession (quickfix.session) is established, and other types of connectiontype are invalid, such as acceptor. Multiple [sessions] can be specified in the configuration file.
(b) Create the Iosessioninitiator through the generated fixsession and incoming eventhandlingstrategy, and save in the initiators (set type of cache).
So what's the difference between fixsession and iosessioninitiator? Please refer to 5, 6.
c) Startup, shutdown client (initiator). Start the initiator by starting the application layer Sessiontimer (refer to 2), and then starting the initiator (iosessioninitiator) of the connection layer. When you close the initiator, close the initiator (iosessioninitiator) of the connection layer, and then close the sessiontimer of the application layer.
4. Quickfix. SessionID: Is the unique identification of the session. SessionID contains beginstring (must), sendercompid (Must), sendersubid (optional), Senderlocationid (optional), targetcompid (required), Targetsubid (optional), targetsubid (optional), Targetlocationid (optional), sessionqualifier (optional). Sessionqualifer is used to differentiate sessions that have the same targetcompid and can only be used in initiator roles. The sessionid.tostring generated readable SessionID strings are composed of: Beginstring:sendercompid/sendersubid/senderlocationid->targetcompid /targetsubid/targetlocationid/sessionqualifier. If an optional value is not set, the default empty string is in the session ID string.
5. Quickfix. Session:session is the most basic abstraction in fix message communication.
A fixsession maintains the self-added serial number of the session's internal message, automatic error recovery, and establishes a communication channel (communication channel) with the communicating partner (counterpart).
b The session is independent of the specific transport layer protocol. Session is
New, the message serial number is set to 1, each communication serial number is increased, until the session is reset (reset). Each session can span multiple transmission connections (not at the same time, but the first network connection is disconnected, then reconnect, although the bottom of the network connection is already new, but the session can remain with the network before the same session).
c) The core logic in Fixsession is in the next () method.
Quickfix. Session.next (): Mainly check the following 8 states, and according to the different state of the response to the operation. Next () is executed periodically in the Sessiontimertask as a routine check session status task.
1 Check whether the session is enabled. What the enabled state represents. Enable is whether the record is currently logged in the session. When logon is performed, the enabled is set to true, and when logout is enabled to false.
If it is not a enabled state and is not a loggedon state, log out of the login normally. End this check and return.
If it is not enabled, but is the LoggedOn status description session last logout or logout message has been sent, on the way back, then check if islogoutsent, if not sent logout, Then send the logout message now.
2 after checking the enabled state of the session, check checksessiontime. It is not necessary to receive every message to check the time stamp of the session, at most 1 seconds to check it, otherwise it is easy to affect performance. If Checksessiontime fails, reset session. (Java.util.concurrent.automatic is used for reset.) Automicboolean as a lock to make reset an atomic operation) reset did something. First callback the user interface Onbeforesessionreset, then generate the logout message and record the Logoutsent status in state to true, then turn off I/O connections and close I/O connections, setting a series of state states. The last ResetState (resets the Messagestore).
3) Check Hasresponder, if there is no responder, then return. For an explanation of responder, please refer to 15.
4) Check islogonreceived, only Receivedlogon is really in the landing state. If you do not receive a logon response, check the state to see if you need to send a logon message (for initiator, you need to send a logon message if you do not have a logon) if you need to send a logon message, Check if logon messages can be sent at this time (in order to prevent recurring logon, there is a frequency limit on sending logon messages), callback user interface Canlogon if it can be sent, determine if the user is allowed to send logon messages, and return directly if not allowed ; if allowed, Generatelogon and send. Returns after the logon message has been sent.
5) Check whether the Getheartbeatinterval is 0. See here, found a quickfix/j bug, searched all the code, did not find as the client initiator anywhere set the state heartbeatinterval. And as server-side acceptor set the state's HeartbeatInterval in Quickfix.mina.acceptor.AcceptorIoHandler.processMessage, the code is as follows: Java code qfsession.setheartbeatinterval (heartbeatinterval);
Qfsession.setheartbeatinterval (HeartbeatInterval);
So when writing a customized initiator, remember to fix this bug, and after the client receives a response from the server-side logon message, remove the value from the Heartbtint field and set it to the Fixsession heartbeatinterval. Settings can refer to the above code.
6) Check state islogouttimedout (Logout message has been sent out, and session has timed out, indicating that the client has actually received the logon message sent by the server, and the client has logout), then disconnect the I/O connection.
7) Check state iswithinheartbeat. Currently found quickfix/j does not set state.withinheartbeat, always use the default initial value false, do not know why. Since this check always returns false, it does not return and continues the next check.
8) Check Stateistimedout (if the distance received heartbeat has exceeded 2.4*heartbeat seconds, it is considered timeout). If the Heartbeatcheck is already timeout, then disconnect the I/O connection and then callback the user's Onheartbeattimeout interface. If you are still not timeout, send testrequest or heartbeat messages as needed.
In STEP1.0.0, in the logon message response, the maxfailint is given, that is, the maximum number of heartbeats that are allowed to fail, after which the other party has timed out and can disconnect and prepare to land again. For the session setting Maxfailint can be maxfailint multiplied by heartbtint multiplied by 1000 to a second, then adding APIs for the session so that the state's Istimedout attribute can be set to operate. And then in this Judge timeout.
The first is Quickfix. SessionState Add a property
Java code Private Long Timeoutms = 24000L;
Private long Timeoutms = 24000L;
Then add a settimeout to a gettimeout, and then add a response setter and getter for the session. Finally, the SetTimeout setting timeout for the session is invoked when a response to the logon message is received.
Quickfix. Session.next (message):
Check the sessiontime first, refresh the timestamp if it is not refreshed for more than 1 seconds, and if the session does not exist, perform reset, reset the session, and return.
It then checks for the beginstring in the header of the message and throws the exception if it does not match the beginstring recorded in SessionID.
Then check Msgtype, if the logon message, according to the version of the different, set the session of the Targetdefaultapplverid. What is the use of Targetdefaultapplverid? Because the FIXT protocol and the previous version of the Fix protocol Applverid have a different composition format, this targetdefaultapplverid is mainly for Quickfix.messagecracker to parse fixt messages.
Then check datadictionaryprovider, according to Customapplverid and Applverid to the data dictionary (applicationdatadictionary and Sessiondatadictionary, For details please refer to 16).
Then, according to the dictionary to verify the message is legitimate, illegal according to the situation of the print warning log or directly throw an exception exit.
Then, depending on the message type, make the appropriate state setting (via Nextlogon,nextheartbeat,nexttestrequest,nextsequencereset,nextlogout), and then call the user back API (in verify).
After the message is processed, nextqueued () is invoked to recursively process the message in the next queue.
Note the Verify (message) function, in which all ordinary messages are invoked to call back the application's Fromapp (Word, SessionID).
Verify-> veriry-> fromcallback ->。 With regard to parsing of messages, the normal message is to parse a string of messages into a message by Quickfix.MessageUtils.parse.
Next (message) is invoked when a singlethreadedeventhandlingstrategy or threadpersessioneventhandlingstrategy process is processed for each received piece of information. is to process the source of the received message.
Quickfix. Session.next (String): is the overloaded function of next (message), parses the input string into a message, and then calls the next (message) processing.
Quickfix. Session.verify (msg, Checktoohigh, Checktoolow): status checks and actions that are made when the msgseqnum of the message received is not the same as expected. For example, if the packet is found, the Resendrequest request is automatically sent or only the alarm log is recorded, if a duplicate msgseqnum is found, if a possible duplicate flag is set in the message, it returns normally, otherwise the message is incorrect, then logout,disconnect.
Quickfix. Session.verify (msg): An overloaded function of verify (MSG, Checktoohigh,checktoolow). Determine Checktoohigh and Checktoolow flags based on whether it is an admin-type message and Checkgapfieldonadminmessage read from the configuration. Checktooxx if it is not an admin message, Checktooxx if Checkgapfieldonadminmessage is true.
Quickfix. Session.dotargettoohigh (Message): Messages received msgseqnum larger than expected, that is, the loss of messages, Check to see if you need Reset or disconnect (as for whether reset or disconnect is specified by the constructor parameter when you create the session, the default is not required, You can also specify the configuration Resetonerror and Disconnectonerror), or reset or disconnect if necessary. If it is not needed, the message is queued (enqueue) into state, and then generated resendrequest and sent out, requesting all messages from the Expectedtargetnum to begin before this message. Thus, the messages stored in the state's MessageQueue are the result of a temporary stored jump after the message ordinal jumps out of the missing message. The messages that are temporarily stored in state are checked and processed after each message is received and processed, such as Next (msg), Nextheartbeat (msg), Nextlogon (msg), Nextreject (msg), Nexttestrequest (msg) The end of these processes examines and processes the messages temporarily stored by the state.
Quickfix. Session.dotargettoolow (Message): Messages received msgseqnum smaller than expected, that is, received a duplicate message, should do things.
Quickfix. Session.parsemessage (StringData): Parses the input string data into a message. is actually called Quickfix.parse (session, StringData) parsing.
Quickfix. Session.send (String messagestring): An API that really sends messages to each other through the bottom of Mina. In quickfix/j, send (msgstring) sends a message through responder, responder finally calls Mina's Iosession.write (object) to write the data to the network.
Responder is through Quickfix. Session.setresponder (Responderresponder) is set when the session is created in Quickfix.mina.initiator.InitiatorIoHandler. For details about responder, please refer to 15.
Quickfix. Session.sendraw (message, int num):
The helper method inside the session, first assemble the incoming message and the message's Msgseqnum, and set the system-related parameters in the message header, such as Beginstring,sendercompid,targetcompid, Sendingtime.
Then, according to the message type callback response user call Backapi, if it is an admin type of message, callback the user's toadmin, non-admin type of message, callback the user's Toapp. If it is a logon message, it is also necessary to reset the Seqnum (the process is: if there is a resetseqnumflag in the message, and the field is set to True) reset. If reset, the first resetstate is taken, and the desired seqnum is set to the Msgseqnum field of the message header. Resetseqnumflag is written to state, regardless of whether it is reset.
Finally, the assembled message call ToString is converted to a string, and then the Send (string) method is invoked to really send the messages to the network.
The entire process needs to be locked sendermsgseqnum until all operations are completed and returned. The roll back will be more efficient if call-back calls fail.
Sendraw is called by Generatexxx in session, such as Generateheartbeat,generatelogon,generatelogout,generatereject, Generatebusinessreject,generateresendrequest,generatesequencereset,generatetestrequest. After generating the corresponding message in these generatexxx, the last call to Sendraw sends the message over the network.
Quickfix. Session.send (message): Two things were done, first removing the Possdupflag and Origsendingtime in the header, and then calling Sendraw to send the message to the network (set Seqnum to 0). The Boolean returned by idea send does not indicate whether the message was successfully sent, simply stating that the message was successfully placed in the Send queue because quickfix/j uses asynchronous IO to send network data by default.
6. Quickfix.mina.initiator.IoSessionInitiator: Use the Transport Layer API provided by Mina to establish and maintain the server
Network Connections at the Transport layer, rather than the network connection of the application tier. These network functions are implemented in a private timertask called Quickfix.mina.initiator.IoSessionInitiator.ConnectTask. Specific implementation features are connected (including normal connections and encrypted SSL connections), reconnection, determining whether to reconnect, handling connection exceptions, starting and stopping Connecttask.
7. Quickfix. Socketinitiator: Use a separate thread to process messages for all sessions. The features provided by Socketinitiator are:
A initialize the initiator with eventhandlingstrategy, then register all sessions managed by this socketinitiator, and then start initiator, The last Call to Eventhandlingstrategy.blockinthread () is in a separate background thread to handle the message of the inserted queue that Sessiontimer received. Start initiator to do is: first start Sessiontimer to listen to the message from the transport layer, if there is no logon first logon, and then after receiving the message callback user code processing messages ; start Reconnecttask to establish and maintain the network connection of the transport layer.
b start the initiator and process the message in the background (Daemon) in another thread.
c) block initiator and process messages in the same thread.
d) Stop initiator. Divided into force stop and non force stop. Force or not force logout all fixsession, stop the initiator of the connection layer, and unregister all sessions managed by this socketinitiator.
e) about how to handle the logic from the underlying message, refer to 11 and 12. Because the so-called processing message here is actually directly or indirectly calling the Singlethreadedeventhandlingstrategy block processing message.
8. Quickfix. Threadedsocketinitiator: Use a separate thread to process messages for each session. Function Reference 7. In addition to the thread working mode, the function is identical to 7.
9. Quickfix. The helper class used in the sessionstate:session and other communication processes. The main function is to store all the States of the session and provide a response API to access these states. States include Heartbeatinterval,heartbeatmillis, whether a heartbeat is needed to determine the role of the application in which the session is located (client initiator or server-side acceptor), Lastreceivedtime,lastsenttime, to obtain logger, to determine whether the client's initiator login message has been issued, to determine whether the server-side acceptor landing message received, to determine whether the need to land, Determine whether the landing is Timeout,messagestore,testrequestcounter, to determine whether the need for testrequest, to determine whether in the timeout state, will receive the message of the team (Enqueue), Out Team (dequeue), lock and unlock send/Receive SequenceNumber, GET, set up the next SequenceNumber, reset (Will sequence empty, start counting again from 1), set up, get the reason for the logout.
Quickfix.mina.EventHandlingStrategy: The interface for policies that handle events for different versions of the fix protocol is the source of the application-level processing message callback interface. When the transport layer message arrives, call this interface onmessage, so to understand, OnMessage is eventhandlingstrategy input, this input comes from the bottom. Getsessionconnector gets the sessionconnector associated with this strategy, that is, to get the input message that is associated with this session to handle the response, usually by layer to the top of the message, Callback the user's function to process the message. Getqueuesize gets the length of the currently processed message queue. Eventhandlingstrategy generally as a member of the Abstractsocketinitiator, when the establishment of Iosessioninitiator to Iosessioninitiator, please refer to 3b). Currently there are two specific implementations in quickfix/j, respectively, Quickfix.mina.SingleThreadedEventHandlingStrategy and Quickfix.mina.ThreadPerSessionEventHandlingStrategy. Please refer to 11, 12 for descriptions of these two specific implementation classes.
Quickfix.mina.SingleThreadedEventHandlingStrategy: Is the core class of quickfix/j processing messages. Single-threaded mode is used when processing messages, even if there are multiple sessions.
A) in order not to block the input, then need a eventqueue to temporarily quickly store all the messages received.
b) OnMessage to the bottom of the incoming message wrapped into sessionmessageevent, first put it into eventqueue.
c) So what's inside the sessionmessageevent. Sessionevent simply wraps the fixsession and message together and provides a way to process the message processmessage. It can be understood that
OnMessage is the input to the underlying message of the event-handling policy, and the ProcessMessage in Sessionevent is the output of the message, which is processed in the future by application-level functions。
D) Getsessionconnector gets the connector that needs to be processed at the application level to handle messages in EventQueue.
e) GetMessage removed from the EventQueue sessionmessageevent to be processed.
f) block is the entry point for processing messages at the application level. Block to determine whether Handlingmessage should continue to be shipped new, if it is to remove the sessionmessageevent from the message queue, call the ProcessMessage to process it.
g) ProcessMessage How to handle the message received. It invokes the Fixsession next method, passing the message to the session, and then relaying the message back to the user's hand by fixsession. Please refer to 5.
h) Perhaps you will notice that the block that handles the message is always invoked in run, and that there is no sleep time, and that it does not stop and does not rest without a word. The CPU will remain 100%. In effect, the secret is that it uses blockingqueue to do the same thing as sleep. In the absence of a message, the loop executes the next loop every second of a rest. How to achieve this effect. The reason is that if there is no message in EventQueue, and the EventQueue sets a blocking timeout of 1000 milliseconds, then the cancellation action waits up to 1000 milliseconds, if not waiting for the message to expire and then quit the wait, execute this loop, If you wait, the message is processed according to the normal process. The great advantage of this is that if there are events in the EventQueue, then the continuous processing, if there is no message, you will rest timeout milliseconds to see.
i) Blockinthread, dealing with sessionmessageevent in a newly initiated background thread
Quickfix.mina.ThreadPerSessionEventHandlingStrategy: This is also the core class of quickfix/j processing messages. Unlike single-threaded mode, the policy starts a new thread for each session to process the message.
A because each session corresponds to a thread, the policy internally requires a processing thread (messagedispatchingthread) reference called Dispatchersmap as the cache to save the response for each session.
b when onmessage receives input from the underlying message, the corresponding processing thread is fetched from the dispatchers according to the input fixsession, and the message is added (enqueue) to the message queue inside the thread to be processed.
c) Each dispatcher (Messagedispatchingthread type) maintains its own message queues, unlike single-threaded mode where messages in message queues are only message, not sessionmessageevent. The logic that handles the message is moved out of the sessionmessageevent in a single thread into the dispatcher.
Quickfix. Datadictionaryprovider: is an interface that provides a data dictionary for the specified sessionprotocol or applicationversion. Getsessiondatadictionary obtains the corresponding data dictionary according to the Beginstring protocol version provided. Getapplicationdatadictionary obtains the data dictionary based on the application VersionID and custom application IDs provided. Application VersionID is determined by the Beginstring field before fixt.1.1. Custom Application ID is an optional value and is not required.
Quickfix. Defaultdatadictionaryprovider: Is the default implementation of the Datadictionaryprovider provided by quickfix/j. In Defaultdatadictionaryprovider, there are two kinds of data dictionaries, one is the data dictionary of transmission, the other is the data dictionary of the application, which is cached in two map respectively. This defaultdatadictionaryprovider was created by the default defaultsessionfactory based on Beginstring when the session was created. Addapplicationdictionary and addtransportdictionary are used to add new data dictionaries to Defaultdatadictionaryprovider respectively. In the current quickfix/j implementation, the dictionary is added in the initialization session in Defaultsessionfactory. For the previous version of the data dictionary, each data dictionary is added to the Fixt data dictionary and the Application Data dictionary.
Quickfix. Responder: This is an interface that is used by the session to send the original fix message (raw fixmessage) or to disconnect the I/O connection to the other. Send (StringData) sends the original fix message, disconnect () disconnects the underlying connection, and getremoteipaddress () provides the other's IP. The responder interface is implemented in the current quickfix/j quickfix.mina.IoSessionResponder. Iosessionresponder.send (string) called Mina's Iosession.write (object) to write data to the network. The default is asynchronous writing. (If synchronous writes display the incoming synchronouswrites as true when the responder is created.) After the synchronization write is actually write (object), take the iosession writefuture and send the join instruction to writefuture. ）
Quickfix. Datadictionary: Provides metadata for different versions of the Fix Protocol (metadata), provides helper methods to help Judge field types, determines types of messages, judges groups, component types, checks message, Field,field, and so on.
There are a total of 3 ways to generate Datadictionary, which can first be read into a configuration file through a system path, or load a inputstream build, or generate from an existing datadictionary copy.
The most complex method in Datadictionary is load, which is based on the definition in Fixxxx.xml to determine whether the XML file is legitimate, and then take the information in each section according to the DTD and deposit it in a predefined map, list, set, The convenient helper method can quickly and accurately search and judge the relevant information.
Quickfix.compositelog: Output log information to multiple logger. Multiple logger are passed in Newcompositelog (log[]logs). However, multiple logger do not have priority, and can not control multiple logger, just log to, while writing to these multiple logger. Compositelog is created by Compositelogfactory.
Other logger in the quickfix/j are:
Quickfix. Screenlog, the log is printed to the screen through System.out, corresponding to the quickfix.screenlogfactory.
Quickfix. FileLog, writes the log to the local file. The corresponding has quickfix. Filelogfactory.
Quickfix. Jdbclog, writes the log to the database. The corresponding has quickfix. Jdbclogfactory.
Quickfix. Slf4jlog, use slf4j wrapper write logs, SLF4J support JDK logging,log4j, and so on.
In addition, most of the log implementations are extends Quickfix.abstractlog. The role of Abstractlog is to add a switch to heartbeat whether logging is logged.
B. The flow of network data in quickfix/j
Connecttask->ioconnector.connect (sockaddress, Iohandler)->mina establish and server-side communication. Receive the network data, Ioconnector trigger the corresponding event, and give the event to Iohandler (Initiatoriohandler) ProcessMessage-> Eventhandlingstrategy.onmessage (Quickfixsession,message) is invoked in ProcessMessage, and the message is externally recalled-> Singlethreadedeventhandlingstrategy.onmessage (quickfixsession, message) will receive messages to the team (EnQueue) to EventQueue-> Singlethreadedeventhandlingstrategy.blockinthread to start a separate background thread, in order to remove message processing from EventQueue, callback to the session-> SessionMessageEvent.quickfixSession.next (message)->quickfixsession.next callback-> per-layer callback according to the Msgtype decision (verify-> Veriry->fromcallback-> Fromadmin/fromapp), callback user processing logic from Fromadmin/fromapp (Msg,sessionid).
