Find the best data storage method

Write by nine days Yan Ling (jtianling) -- blog.csdn.net/vagrxie

Discuss newsgroups and documents

There is always a lot of related data to be stored in the game, such as the level data generated by the editor, the configuration of characters and item attributes, etc, now, not everyone has designed a data format for themselves. programmers are getting more and more lazy, and everyone is thinking about a set of file formats, a set of parsing, processing code can eat all the programs. the following discussions, including the advantages and disadvantages, refer only to the premise background for the storage of game data (or edited data only). This article is for reference only in other fields, the use of the game field is only a family of words, of course, for reference only.
First, let's look at the traditional data storage solution.

Traditional data storage solution Excel Solution

The biggest advantage is that almost everyone can change and do not need to learn more. Do this, not to mention planning, even the Secretary should at least Excel?
Disadvantage: You need to write a lot of Excel parsing code by yourself, which is helpless.
In addition, if you do not want to write a set of parsing code for each excel file, it is not easy to think of a general Excel format with good scalability.
Even if the designed Excel format is highly scalable, when the data is slowly complicated, you will find that things are becoming more and more complex ..... I almost started to use Excel as an associated database. sometimes, modifying Excel Data with strong relevance is more complicated than modifying code ..... at that time, you will find that excel is not enough.
Most importantly, Excel (at least version 03) is binary data. version control tools such as SVN and mercurial (this is what everyone uses in the project) do not support merge, in this way, every time many people change, it will inevitably lead to a conflict. Then, manually rewrite it.
However, if you encounter a translation group that only knows how to use Excel as our company said, then you will suffer and you will be helpless. this is also the advantage of Excel ........ of course, from the perspective of the program, that is the biggest drawback.

Database Solution

The biggest advantage of the database solution is that the scalability is extremely powerful, and the scalability of the database solution is almost unlimited. No matter how large the data volume is, it will not always appear much in the database, associated tables can always minimize redundant data and are so clear. as the front-end of modern databases is getting better and better, modifying data in databases does not always require SQL statements. most importantly, you can use SQL statements to perform the most complex queries you need, or modify some related data in batches.
Of course, no solution is perfect. The database solution is indeed powerful, but it also has a powerful price. First of all, database experts in the project can design enough tables, and write complex multi-nested select statements. Some people in the project know how to use their own database interface APIs in their own languages. What is depressing is that, the API design for various databases is not the same ...... generic API design is often at the cost of low efficiency .... of course, these are all far away from the original server experience ........ if we only use databases as the data storage method for general games, the disadvantages are not so exaggerated. Moreover, we can always use open-source solutions such as MySQL and sqllite. there are always available APIs in various languages.
In addition to the above, there is still a problem. Even if we only use the database as a normal game data storage method, we design a set of tables for each requirement and write a bunch of code to get data from the database, in addition, to publish binary data, you have to write a set of code that reads data from the database and serializes the data into binary data, it is also essential to parse binary code. powerful database solutions pile up large amounts of manual code.

Modern Data Storage Solutions

Because the traditional solution has its inherent disadvantages, Excel is difficult to store more and more game data for more and more games. Instead, it uses the database solution to store game data (only to edit data, excluding online game running data) is too bloated. is there a better way? Let's first look at what we want:
1. There is a text storage method, which facilitates debugging and modification during the development period. You can also intuitively view the changes and Version Control Merge during the development period.
2. There is a binary storage method, which can be used after the release to reduce the parse time, and there is a very simple solution to switch from the text mode to the binary mode.
3. parse: serialization code compilation should be simple enough, at least not as the data size expands to a point that is hard to grasp.
4. multi-language support, which you can always use in your preferred language
5. to establish associations between data, nesting relationships must be simple enough to easily express rich content, instead of creating another set of data to represent the Association (as needed in Excel)

Is there a solution that can meet all the above requirements? For more information, see.
Because of my limited capabilities, Here we only use XML, JSON, and Google protobuf, which are commonly used as examples for comparison and reference for choosing the best method. (A solution named thrift was also found during the data search process.
Compared with Google protobuf, it indicates that it is much more powerful than Google protobuf, because it is rarely used and I am not going to try it. This information is provided here for reference only)

XML Scheme

Currently, the XML solution in game data is widely used. Is it actually an industrial standard? At least I feel like this. Even after Office 2007, the general format is XML. is it because Web development is so popular that everyone enjoys the markup language with curly brackets? I don't know. However, XML does have its advantages. In Wikipedia, there is such a description:
As of 2009<sup>[Update]

, Hundreds of XML-based versions ages have been developed,[5]

Including RSS
, (Standard) "href =" http://en.wikipedia.org/wiki/Atom_%28standard%29 "> Atom
, Soap
, And XHTML
. XML-based formats have become the default for most office-productivity tools, including office "href =" http://en.wikipedia.org/wiki/Microsoft_Office "> Microsoft Office
(XML "href =" http://en.wikipedia.org/wiki/Office_Open_XML "> office open XML
), OpenOffice.org
(Opendocument
), And computer "href =" http://en.wikipedia.org/wiki/Apple_Computer "> Apple
'S iwork
.[6]

Let's take a look. There are a lot of applications.
First, requirements 1, 3, 4, and 5 of the above five points are perfectly satisfied. XML is a data storage format for people to read. Although people who like angle brackets (such as me) will dislike it, and because of its popularity, the XML library for parsing is also very rich, all types of languages must support xml. xml also provides excellent support for nesting and association. Its self-description allows you to see what XML is describing. XML processing is a string that supports Unicode. An interesting example is provided on Wikipedia:
Version
= "1.0"
Encoding
= "UTF-8"
?>


<Commandid>

When there are already too many threads, there are already too many threads</sup>