Preface I read sphtracing's code for reading configuration files. I have a question in my mind: Why should sphtracing build its own wheels? Is it because no upgrade or maintenance has been performed in the sphinx all the time? No matter how much it is, let's take a look at how sphpilot analyzes the configuration file and stores the configuration file. The configuration file rules are as follows:
Preface I read sphtracing's code for reading configuration files. I have a question in my mind: Why should sphtracing build its own wheels? Is it because no upgrade or maintenance has been performed in the sphinx all the time? No matter how much it is, let's take a look at how sphpilot analyzes the configuration file and stores the configuration file. The configuration file rules are as follows:
Preface
After reading the code for reading the configuration file of sphinx, I have a question: why does sphinx need to build its own wheels?
Is it because no upgrade or maintenance has been performed in the sphinx all the time?
No matter how much it is, let's take a look at how sphpilot analyzes the configuration file and stores the configuration file.
Configuration File rules
The following is a simple sphinx configuration file with incremental indexes.
Inc_source inherits the base_source source.
Inc_index inherits inc_index. Then, the indexer and searchd options are configured.
source base_source { type = mysql sql_host = 127.0.0.1 sql_user = test sql_pass = test sql_db = d_test sql_port = 3306 sql_query_pre= SET NAMES utf8 sql_query = select c_id, c_title, c_content, c_comments, c_mtime FROM t_post; sql_attr_uint = c_comments sql_attr_string = c_title sql_field_string = c_content sql_field_string = c_mtime}source inc_source : base_source{ sql_query_pre = SET NAMES utf8 sql_query = select c_id, c_title, c_content, c_comments FROM t_post where c_mtime >=(SELECT c_mtime FROM t_post_inc where c_id = 1) ; }index base_index { source = base_source path = var/data/base_index docinfo = extern charset_type = zh_cn.utf-8 charset_dictpath = etc/ ngram_len = 0}index inc_index : base_index{ source = inc_source path = var/data/inc_index}indexer{ mem_limit = 32M}searchd{ listen = 9312 listen = 9306:mysql41 log = var/log/searchd.log query_log = var/log/query.log pid_file = var/log/searchd.pid}Configuration file data structure
For the preceding configuration file, we can use a json representation.
That is, a series of recursive key-values.
If the entire file is a root, the first-level son has four sons: source, index, indexer, and searchd.
Then the son of index and source is an array, and each position of the array is an object, because Multiple indexes and source can be configured.
The son of indexer and searchd is an object. The objects mentioned above are a series of key-value leaf nodes.
For the root node, sphinx uses a hash for storage. In fact, all nodes use hash for storage.
/// config (hash of section types)typedef SmallStringHash_T < CSphConfigType > CSphConfig;
The second-level node is another hash object.
/// config section type (hash of sections)typedef SmallStringHash_T < CSphConfigSection > CSphConfigType;
The CSphConfigSection level requires two types: leaf nodes and arrays.
Instead of using a simple hash object, You Need To encapsulate a layer based on the hash object.
In fact, sphenders encapsulate layers instead of layers.
/// config section (hash of variant values)class CSphConfigSection : public SmallStringHash_T < CSphVariant >;/// small hash with string keystemplate < typename T >class SmallStringHash_T : public CSphOrderedHash < T, CSphString, CSphStrHashFunc, 256 >;/// simple dynamic hash/// implementation: fixed-size bucket + chaining/// keeps the order, so Iterate() return the entries in the order they was inserted/// WARNING: slow copytemplate < typename T, typename KEY, typename HASHFUNC, int LENGTH > class CSphOrderedHash;
The last layer finally saw the base class.
Json and hash table
We can see that all nodes inherit from the SmallStringHash_T template class.
This class helps us implement the key-value storage method.
Then how did he implement it?
Let's talk about the requirements first. We have some key-value columns, and then we can quickly add and find them.
How does this look familiar?
By the way, isn't that exactly what hash table does?
A few days ago, I wrote two articles: hash table Research and Implementation and memcached source code reading.
Is it necessary to write another hash table for sphinx this time?
To know how a key-value is stored, you just need to check its add function.
Next let's take a look at the add operation of sphinx.
/// add new entry/// returns true on success/// returns false if this key is already hashedbool Add ( const T & tValue, const KEY & tKey ){ unsigned int uHash = ( (unsigned int) HASHFUNC::Hash ( tKey ) ) % LENGTH; // check if this key is already hashed HashEntry_t * pEntry = m_dHash [ uHash ]; HashEntry_t ** ppEntry = &m_dHash [ uHash ]; while ( pEntry ){ if ( pEntry->m_tKey==tKey )return false; ppEntry = &pEntry->m_pNextByHash; pEntry = pEntry->m_pNextByHash; } pEntry = new HashEntry_t; // do other thing}
Well, seeing the loop above, I think of the cover picture at the top of the hash table read by memcached source code.
Typical chain table hash.
What does it mean?
If the hash conflicts, store it as a linked list.
Here we will not talk about the hash table of sphinx.
Then we can use the hash-table recursion to implement our own json.
The unique json Storage types are complicated, while our types are relatively simple. The values of leaf nodes are strings and numbers.
However, this affects the functions of json.
Configuration file parser
Here, we will find that storing configuration information is not a problem. The problem is how to convert this configuration file into the data structure we store.
In fact, our configuration file is quite complicated. Of course, I compared it with the ini configuration file.
This is basically the form of configuration files, and usually several layers need to be nested inside.
No matter how complicated the configuration file is, it is the same for our configuration file parser.
Why?
As a Computer Major, everyone has learned how to compile when they go to college.
The first few pieces of the course are about the basic principle of the parser, namely, the automatic machine.
We define a set of rules. The configuration file is configured according to this set of rules, and the parser can parse according to this set of rules.
After talking about this, let's talk about the status of the automatic machine.
The sphinx configuration file can be expressed in the following states.
enum { S_TOP, S_SKIP2NL, S_TOK, S_TYPE, S_SEC, S_CHR, S_VALUE, S_SECNAME, S_SECBASE, S_KEY }
S_TOP is the starting state.
S_SKIP2NL is the annotation state, and only line annotations are supported in the configuration file. Therefore, the content following the annotation identifier is often ignored.
Of course, the first line is of special use, and sphinx is also of special use.
However, I have not found this document. The sphinx source code indicates that a specified program can be executed, and the parameter is the configuration file.
We don't care about this anymore. Let's assume there is no such item.
Generally, after blank space is ignored in the S_TOP state, the first problem is one of the four strings: source, index, searchd, and indexer.
Therefore, S_TOP will first call S_TOK to read a word (token) and then go to the S_TYPE status.
S_TYPE is split into two states based on the read string.
Here, source and index are switched to S_SECNAME, while indexer, searchd, and common are switched to S_SEC.
Why?
The reason is that there is a name after source and index, while others have no name, and they are directly key-value.
S_SEC status is simple. Let's first look at the transfer of S_SEC status.
Of course, before transferring to the S_SEC status, we need to read a left braces "{" and read a definite word from the status S_CHR.
In S_SEC, what we need to do is read a key-value, where key-value is separated by equal signs.
So for the first time we encounter non-null characters, We need to read the key first, so S_SEC is transferred to the state S_KEY.
The end character "}" indicates that the configuration is complete.
The S_KEY status is a token, so we need to call the S_TOK status to read this token.
In S_KEY, we need to first read a definite equal sign "=" (State S_CHR), then read the value, so it is transferred to the state S_VALUE.
In the S_VALUE state, we need to first store the read value, and then call the function to add the key-value when an Terminator is encountered.
At this time, we have completed a key-value for reading the S_SEC status until the read end character "}" is returned to the S_TOP status.
When S_TOP is switched to S_SECNAME, We need to read the configuration name first.
So we first turn to the S_TOK status to read a token.
Then there will be two situations: one is the starting symbol "{", and we turn to the State S_SEC.
In another case, we encounter ":", which means that this configuration should be integrated with a previous configuration.
Therefore, we need to first read the name of a base, that is, to the Status S_SECBASE.
S_SECBASE first switches to S_TOK to read the name of a base.
Then, the configuration named base is copied to the current configuration.
Then you can use S_CHR to read the start character "{" and turn to the State S_SEC.
In this way, the state automation is complete.
For details, refer to the following figure:
The simple status is as follows:
#define LOC_PUSH(_new) { eStack[iStack++] = eState; eState = _new; }#define LOC_POP() { eState = eStack[--iStack]; }#define LOC_BACK() { p--; }for (;; p++) { // if this line is over, load next line // handle S_TOP state if (eState == S_TOP) { iToken = 0; LOC_PUSH(S_TYPE); LOC_PUSH(S_TOK); LOC_BACK(); continue; } // handle S_SKIP2NL state if (eState == S_SKIP2NL) { LOC_POP (); p = pEnd; continue; } // handle S_TOK state if (eState == S_TOK) { if (!sphIsAlpha(*p)) { LOC_POP (); sToken[iToken] = '\0'; iToken = 0; LOC_BACK(); continue; } if (!iToken) { sToken[0] = '\0'; } sToken[iToken++] = *p; continue; } // handle S_TYPE state if (eState == S_TYPE) { if (IsPlainSection(sToken)) { if (!AddSection(sToken, sToken))break; sToken[0] = '\0'; LOC_POP(); LOC_PUSH(S_SEC); LOC_PUSH(S_CHR); iCh = '{'; LOC_BACK(); continue; } if (IsNamedSection(sToken)) { m_sSectionType = sToken; sToken[0] = '\0'; LOC_POP (); LOC_PUSH(S_SECNAME); LOC_BACK(); continue; } } // handle S_CHR state if (eState == S_CHR) { LOC_POP (); continue; } // handle S_SEC state if (eState == S_SEC) { if (*p == '}') { LOC_POP (); continue; } if (sphIsAlpha(*p)) { LOC_PUSH(S_KEY); LOC_PUSH(S_TOK); LOC_BACK(); iValue = 0; sValue[0] = '\0'; continue; } } // handle S_KEY state if (eState == S_KEY) { LOC_POP (); LOC_PUSH(S_VALUE); LOC_PUSH(S_CHR); iCh = '='; LOC_BACK() continue; } // handle S_VALUE state if (eState == S_VALUE) { if (*p == '\n') { AddKey(sToken, sValue); iValue = 0; LOC_POP (); continue; } if (iValue < iValueMax) { sValue[iValue++] = *p; sValue[iValue] = '\0'; } continue; } // handle S_SECNAME state if (eState == S_SECNAME) { if (!sToken[0]) { LOC_PUSH(S_TOK); LOC_BACK(); continue; } AddSection(m_sSectionType.cstr(), sToken); sToken[0] = '\0'; if (*p == ':') { eState = S_SECBASE; continue; } if (*p == '{') { eState = S_SEC; continue; } } // handle S_SECBASE state if (eState == S_SECBASE) { if (!sToken[0]) { LOC_PUSH(S_TOK); LOC_BACK() ; continue; } // copy the section LOC_BACK(); eState = S_SEC; LOC_PUSH(S_CHR); iCh = '{'; continue; }}
Original article address: sphinx source code reading json, hash table configuration analyzer, thanks to the original author for sharing.