Java read LEVEL-1 market dbf Files Extreme Optimization (1)

The recent construction of a project to achieve market access and distribution, the need to achieve the ultimate low latency characteristics, which is very important for the security system. Access to the market source is configurable, either LEVEL-1 or Level-2 or other third-party sources. Although LEVEL-1 market is not Level-2 fast, but as a system to support the market source, we still need to optimize it, so that from the file read, to the user through the socket to receive the market, end-to-end latency as low as possible. This paper mainly introduces the extreme optimization scheme of LEVEL-1 market dbf file reading. Believe that the other DBF file read should also have reference significance.

LEVEL-1 market is by the market station, timed every few seconds to the DBF file (Shanghai is show2003.dbf, Shenzhen is sjshq.dbf) update again, with the new market replaced the old. Our goal is to read the files into memory in the shortest amount of time after the new files have been updated, convert each row into objects, and convert each column to the corresponding data type.

We have adopted 6 optimization methods altogether.

Optimization one: Using memory hard disk (RamDisk)

Memory hard disk can greatly improve the file read and write speed, level-1 market reading and writing is the application of memory hard disk is a perfect situation:

1, we can put the price of the market station file address on the memory hard disk. This can accelerate the speed of writing files at the market station.

2, the system again from the memory hard disk read, but also to speed up the reading speed.

3, the memory hard drive loses the file, here we basically do not care about this shortcoming, because the market document is originally temporary, if has the need of persistence, most of the memory hard disk also supports persistent function.

Windows has a lot of virtual memory hard disk software (the system is deployed on Windows Server, because the market station is on windows). Performance is still different, I mainly based on the following article "RAM Disk software benchmarked for fastest Read and Write speed", I did a little bit of testing, chose the bond disc this software. Its address is http://www.bonddisc.com/, the use is very simple, here is not detailed introduction.

Test down, the use of RAMDisk read is not much faster than from the ordinary hard disk reading, probably because the market dbf file itself is very small, only 1~2m, and we use the NiO form of the file read itself is relatively fast. But the use of RAMDisk is still very meaningful--

1, because the market station directly to the DBF file written to the memory hard disk, can speed up a lot.

2, if the hard disk is busy, because the hard disk is serial, the use of memory hard disk in this case can avoid bottlenecks, maintain stable read low latency.

Optimization two: Using Jnotify, with notifications instead of polling

As the market will continue to update the market DBF file, our system needs to detect once the market file is updated, immediately read. The traditional strategy is to constantly poll the status of the market file, if the last modification date of the market document (or plus the file size) changes, it is considered that the file is updated. But this approach is inefficient, time-delay is high and unstable. Let's say that even if the polling time is set to 10ms once (which means polling 100 times in 1 seconds), the average latency is 5ms.

We can use notifications instead of polling. Here we use the Jnotify library, the library is:

http://jnotify.sourceforge.net/

The Jnotify library supports Windows,linux and MacOS, allowing us to monitor a folder and initiate a callback notification when a file under this folder is changed or deleted. The code examples are as follows:

 Public voidAddwatcher (String hangqingfolder, String hangqingfile)throwsException {//Watch mask, specify events//or Jnotify.file_any for all events.        intMask =//jnotify.file_created | //jnotify.file_deleted |jnotify.file_modified;//we only need to register the modification event//jnotify.file_renamed; //watch subtree?        BooleanWatchsubtree =false; //Add actual watchWatchid= Jnotify.addwatch (Hangqingfolder, Mask, Watchsubtree,NewJnotifylistener () { Public voidFilerenamed (intWD, String RootPath, String oldname, String newName) {                //Do nothing .            }             Public voidFilemodified (intWD, String RootPath, string name) {                if(!hangqingfile.equalsignorecase (name))//If the other file is modified, ignore the                    return; Readhangqingfile (Hangqingfolder+ File.separator +hangqingfile); }             Public voidFiledeleted (intWD, String RootPath, string name) {                // do nothing            }             Public voidFilecreated (intWD, String RootPath, string name) {                // do nothing            }        }); }

The above code:

1, we just need to monitor the file modification, so just set mask = jnotify.file_modified

2, no need to recursively monitor subdirectories, set Watchsubtree = False

3, because the monitoring is the folder, not the file, in the Filemodified method, we want to determine whether the changes we care about the file (i.e. the market file), if not, then ignore. If it is, call Readhangqingfile to begin reading.

Jnotify is implemented based on the operating system API, which is implemented using JNI, and therefore contains. dll files and. So files in addition to jar files. When developing with eclipse, we need to specify the directories for these local libraries, as shown in:

When deploying, you need to place the local library under the execution root, or specify the location of the local library with -djava.library.path=/native/library/path .

Using Jnotify, with (operating system-based) notifications instead of polling, you can find that files are updated very quickly, based on the test time <1ms (I think it should be far less than 1ms, but because the file modification time unit is milliseconds, there is no way to measure more accurately).

Cond...

Binhua Liu original article, reprint please specify the original address http://www.cnblogs.com/Binhua-Liu/p/5609396.html

Java read LEVEL-1 market dbf Files Extreme Optimization (1)