Java read Level-1 quotes dbf File optimization (1), level-1dbf

Java read Level-1 quotes dbf File optimization (1), level-1dbf

Recently, a project was constructed to achieve market access and distribution, which requires extreme low latency. This is very important for the securities system. The access line source can be configured, either Level-1 or Level-2 or another third-party source. Although the Level-1 market is not as fast as Level-2, we still need to optimize it as the line source supported by the system so that the user can read the file and receive the quotation through socket, the end-to-end latency is as low as possible. This article describes the optimal solution for reading level-1 dbf Files. I believe that reading other dbf Files should also be of reference significance.

 

Level-1 market is a small market site. It regularly updates the dbf File (show2003.dbf in Shanghai and sjshq. dbf in Shenzhen) every few seconds and replaces the old one with the new market. Our goal is to read the file to the memory in the shortest time after the new file is updated, convert each row to an object, and convert each column to the corresponding data type.

 

We have adopted six optimization methods in total.

 

Optimization 1: Use a memory hard drive (RamDisk)

Memory hard disks can greatly improve the speed of reading and writing files. level-1 reading and writing is an excellent situation for memory hard disks:

1. We can configure the address of the quote file on the memory hard disk. This will accelerate the speed of writing files to small sites.

2. The system reads data from the memory hard disk, which can speed up reading.

3. files will be lost when the memory hard disk powers down. Here we basically don't care about this shortcoming, because the market file is originally temporary. If there is a need for persistence, most memory hard disks also support the persistence function.

 

Windows has a lot of software with virtual memory hard disks (the system is deployed on a windows Server, because the quotation station is on windows ). Performance is still poor. I chose Bond Disc based on the following Article 12 RAM Disk Software Benchmarked for Fastest Read and Write Speed. The address is http://www.bonddisc.com/. it is easy to use and will not be detailed here.

 

After the test, reading data using RamDisk is not much faster than reading data from a common hard disk. It may be because the dbf file itself is very small, only 1 ~ 2 M, and we use NIO to read files faster. But it makes sense to use RamDisk --

1. Because the website directly writes the dbf file to the memory hard disk, it can speed up a lot.

2. If the hard disk is serial when it is busy, the memory hard disk can be used to avoid bottlenecks and maintain stable read and low latency.

 

Optimization 2: Use JNotify instead of polling with notifications

Because the quotation site will constantly update the quotation dbf file, our system needs to detect that once the quotation file is updated, it will be read immediately. The traditional strategy is to keep polling the status of the quote file. If the last modification date of the quote file (or the file size is added) is changed, the file is considered to be updated. However, this method is inefficient, with high latency and unstable. Assume that the average latency is 5 ms even if the polling time is set to 10 ms once (which means 100 polling times in one second.

 

We can use notifications instead of polling. Here we use the JNotify library, which is:

Http://jnotify.sourceforge.net/

The JNotify Library supports Windows, Linux, and MacOS. We can monitor a folder and initiate a callback notification when the files in this folder are added, deleted, and modified. The sample code is as follows:

Public void addWatcher (String hangqingFolder, String hangqingFile) throws Exception {// watch mask, specify events you care about, // or JNotify. FILE_ANY for all events. int mask = // JNotify. FILE_CREATED | // jpolicy. FILE_DELETED | jpolicy. FILE_MODIFIED; // we only need to register the modification event // jpolicy. FILE_RENAMED; // watch subtree? Boolean watchSubtree = false; // add actual watch watchId = JNotify. addWatch (hangqingFolder, mask, watchSubtree, new jpolicylistener () {public void fileRenamed (int wd, String rootPath, String oldName, String newName) {// do nothing .} public void fileModified (int wd, String rootPath, String name) {if (! HangqingFile. equalsIgnoreCase (name) // if other files are modified, the return; readHangqingFile (hangqingFolder + File. separator + hangqingFile);} public void fileDeleted (int wd, String rootPath, String name) {// do nothing} public void fileCreated (int wd, String rootPath, String name) {// do nothing }});}

 

The above code:

1. We only need to monitor file modifications, so we only need to set mask = jpolicy. FILE_MODIFIED

2. Do not recursively monitor sub-directories. Set watchSubtree to false.

3. Because folders are monitored, rather than files, In the fileModified method, we need to determine whether the file we are interested in is modified (that is, the market file). If not, ignore it. If yes, call readHangqingFile to start reading.

 

JNotify is implemented based on the operating system API, that is, using JNI. Therefore, in addition to jar files, it also contains. dll files and. so files. When using eclipse for development, we need to specify the directories of these local libraries, as shown in:

 

During deployment, you need to put the local library in the execution root directory, or use-Djava. library. path =/native/library/pathSpecifies the location of the local database.

 

Using JNotify and using (OS-based) Notifications instead of polling, you can quickly find files updated. According to the test time <1 ms (I think it should be far less than 1 ms, however, since the file is modified in milliseconds, there is no way to make more precise measurements ).

 

To be continued...

 

Binhua Liu Original article, reprint please indicate the original address http://www.cnblogs.com/Binhua-Liu/p/5609396.html