TinyDBF-200 line write DBF parser

Due to the work relationship, we need to read the DBF file and find some open-source software for DBF. Either it is too large and tens of thousands of lines are not moving, or the function is faulty, encoding and length. In short, there is no great deal. In desperation, my old man was so angry that he dared to write it by himself. The result is used only.

Due to the work relationship, we need to read the DBF file and find some open-source software for DBF. Either it is too large and tens of thousands of lines are not moving, or the function is faulty, encoding and length. In short, there is no great deal. In desperation, my old man was so angry that he dared to write it by himself. The result is used only.

Preface

Due to the work relationship, you need to read the DBF file and find some open-source software for DBF reading. Either it is too large, and tens of thousands of lines are not moved, or there is a functional problem, encoding, length. In short, it is not found a very nice one. In desperation, my old man was so angry that he dared to write it by himself. The result is handled with less than 300 lines of code. Of course, it is not the only goal, but also elegant and concise. Let's take a look at the simple design and implementation together with me.

Before coding, let's first introduce DBF. This DBF is an old thing and has appeared in the DOS era for quite a while. Later, with the application of large databases, it gradually declined, but due to its concise and easy-to-use features, it is still applied in a large number of data exchanges. However, in the development process, many versions have been formed. Different versions have different structures, which determines that their parsing programs are also different.

Today, I only implemented the parsing of Foxbase/DBaseIII, but I am also ready to expand various other versions.

Interface Design

There are two classes in total. One interface, Field and Header are two simple POJO classes that define the file Header and Field-related information respectively.

Reader interface is an interface for reading DBF Files. It mainly defines methods for obtaining file types, encoding, fields, and recording movement.

Code Implementation

First, implement the abstract class of Reader

Public abstract class DbfReader implements Reader {protected String encode = "GBK"; private FileChannel fileChannel; protected Header header; protected List
 
  
Fields; private boolean recordRemoved; int position = 0; static Map
  
   
ReaderMap = new HashMap
   
    
(); Static {addReader (3, FoxproDBase3Reader. class);} public static void addReader (int type, Class clazz) {readerMap. put (type, clazz);} public static void addReader (int type, String className) throws ClassNotFoundException {readerMap. put (type, Class. forName (className);} public byte getType () {return 3;} public String getEncode () {return encode;} public Header getHeader () {return header;} public List
    
     
GetFields () {return fields;} public boolean isRecordRemoved () {return recordRemoved;} public static Reader parse (String dbfFile, String encode) throws IOException, IllegalAccessException, instantiationException {return parse (new File (dbfFile), encode);} public static Reader parse (String dbfFile) throws IOException, IllegalAccessException, InstantiationException {return parse (new File (dbfFile ), "GBK");} public static Reader parse (File dbfFile) throws IOException, IllegalAccessException, InstantiationException {return parse (dbfFile, "GBK");} public static Reader parse (File dbfFile, string encode) throws IOException, IllegalAccessException, InstantiationException {RandomAccessFile aFile = new RandomAccessFile (dbfFile, "r"); FileChannel fileChannel = aFile. getChannel (); ByteBuffer byteBuffer = ByteBuffer. allocate (1); fileChannel. read (byteBuffer); byte type = byteBuffer. array () [0]; Class
     
      
ReaderClass = readerMap. get (int) type); if (readerClass = null) {fileChannel. close (); throw new IOException ("unsupported file type [" + type + "]. ");} DbfReader reader = (DbfReader) readerClass. newInstance (); reader. setFileChannel (fileChannel); reader. readHeader (); reader. readFields (); return reader;} public void setFileChannel (FileChannel fileChannel) {this. fileChannel = fileChannel;} protected abstract void readFields () throws IOException; public void moveBeforeFirst () throws IOException {position = 0; fileChannel. position (header. getH EaderLength ();}/*** @ param position starts from 1 * @ throws java. io. IOException */public void absolute (int position) throws IOException {checkPosition (position); this. position = position; fileChannel. position (header. getHeaderLength () + (position-1) * header. getRecordLength ();} private void checkPosition (int position) throws IOException {if (position> = header. getRecordCount () {throw new IOExce Ption ("the expected number of records is" + (this. position + 1) + ", exceeding the actual number of records:" + header. getRecordCount () + ". ") ;}} Protected abstract Field readField () throws IOException; protected abstract void readHeader () throws IOException; private void skipHeaderTerminator () throws IOException {ByteBuffer byteBuffer = ByteBuffer. allocate (1); readByteBuffer (byteBuffer);} public void close () throws IOException {fileChannel. close ();} public void next () throws IOException {checkPosition (position); ByteBuffer byteBuffer = ByteBuffer. allocate (1); readByteBuffer (byteBuffer); this. recordRemoved = (byteBuffer. array () [0] = '*'); for (Field field: fields) {read (field);} position ++;} public boolean hasNext () {return position 
    
   
  
 

This class is the largest class. It is worth noting that there are several static methods: addReader and parse,

AddReader is used to add new types of Reader, and parse is used to parse Files.

The execution process of parse is to first read the first byte and determine whether there is a corresponding parsing implementation class. If there is a corresponding parsing implementation class, if there is no, the error declaration is not supported.

The following is a simple implementation class. The following is the parser of FoxproDBase3:

public class FoxproDBase3Reader extends DbfReader {    protected void readFields() throws IOException {        fields = new ArrayList
 
  ();        for (int i = 0; i < (header.getHeaderLength() - 32 - 1) / 32; i++) {            fields.add(readField());        }    }    public byte getType() {        return 3;    }    protected Field readField() throws IOException {        Field field = new Field();        ByteBuffer byteBuffer = ByteBuffer.allocate(32);        readByteBuffer(byteBuffer);        byte[] bytes = byteBuffer.array();        field.setName(new String(bytes, 0, 11, encode).trim().split("\0")[0]);        field.setType((char) bytes[11]);        field.setDisplacement(Util.getUnsignedInt(bytes, 12, 4));        field.setLength(Util.getUnsignedInt(bytes, 16, 1));        field.setDecimal(Util.getUnsignedInt(bytes, 17, 1));        field.setFlag(bytes[18]);        return field;    }    protected void readHeader() throws IOException {        header = new Header();        ByteBuffer byteBuffer = ByteBuffer.allocate(31);        readByteBuffer(byteBuffer);        byte[] bytes = byteBuffer.array();        header.setLastUpdate((Util.getUnsignedInt(bytes, 0, 1) + 1900) * 10000 + Util.getUnsignedInt(bytes, 1, 1) * 100 + Util.getUnsignedInt(bytes, 2, 1));        header.setRecordCount(Util.getUnsignedInt(bytes, 3, 4));        header.setHeaderLength(Util.getUnsignedInt(bytes, 7, 2));        header.setRecordLength(Util.getUnsignedInt(bytes, 9, 2));    }}
 

Test Cases

public class DbfReaderTest {    static String[] files = {"BESTIMATE20140401", "BHDQUOTE20140401"};    public static void main(String[] args) throws IOException, IllegalAccessException, InstantiationException {        for (String file : files) {            printFile(file);        }    }    public static void printFile(String fileName) throws IOException, InstantiationException, IllegalAccessException {        Reader dbfReader = DbfReader.parse("E:\\20140401\\" + fileName + ".DBF");        for (Field field : dbfReader.getFields()) {            System.out.printf("name:%s %s(%d,%d)\n", field.getName(), field.getType(), field.getLength(), field.getDecimal());        }        System.out.println();        for (int i = 0; i < dbfReader.getHeader().getRecordCount(); i++) {            dbfReader.next();            for (Field field : dbfReader.getFields()) {                System.out.printf("%" + field.getLength() + "s", field.getStringValue());            }            System.out.println();        }        dbfReader.close();    }}

We can see that the last use is very simple.

Code statistics

The total number of lines of code is 282. If the number of lines is removed from the import and interface declaration, there are about 200 lines of code that actually work:

Summary

The above not only shows how to parse the DBF file, but also shows how to properly balance the current needs and future extensions.

For example, to implement support for another standard DBF file, call DbfParser. addReader (xxxReader) after simple implementation, just like the FoxproDBase3Reader class above );

A good design needs to avoid over-design and make it too complicated. At the same time, we should also consider future changes and extensions to avoid the need to move here when new demands come, there are changes and changes in the structure, and pay attention to observe the DRY principle. It can be said that if there is a lot of repetition in the program, there must be structural design problems.

All the code can be seen in the following connection:

Https://code.csdn.net/tinygroup/tiny/tree/master/framework/org.tinygroup.dbf