Recently, I used python to write a simple compiler for parsing protobuf files. I feel that ply is simple and convenient for lexical analysis and syntax analysis. Take advantage of the waste of heat and be clear-headed. write down your summary and experiences to facilitate your reference and use of pythoner.
Introduction
Recently, I used python to write a simple compiler for parsing protobuf files. I feel that ply is simple and convenient for lexical analysis and syntax analysis. Take advantage of the waste of heat and be clear-headed. write down your summary and experiences to facilitate your reference and use of pythoner.
Use ply
Introduction
If you are not engaged in compiler or parser development, you may have never heard of ply. Ply is a python-based lex and yacc, and its authors are the famous Python Cookbook and 3rd Edition authors. Some of my friends may be wondering, how do I need to write a compiler for a single business development? as programmers say, the central government decides to try new things more. It is also helpful to understand the syntax parsing posture and parse complicated logs or mathematical formulas.
For children's shoes that do not have a foundation for compilation, it is strongly recommended that you understand some basic concepts related to grammar. Parsing techniques strongly recommended by wheel brother and compiling the Dragon and Tiger whale books are not suitable for beginners. here, we recommend Hu Lunjun's compilation principles (Electronic Industry Press ), there are many examples of concepts, which are suitable for beginners. Of course, we do not need to study it in depth. knowing the concepts and methods of lexical analysis and syntax analysis can make ply easy to use. Link: http://www.pchou.info/open-source/2014/01/18/52da47204d4cb.html
To help you get started, let's take solving the multivariate one-time equations as an example to explain how to use ply.
Example
The input is an equation in the format of x + 4y-3.2z = 7. to make the example as simple as possible, make the following restrictions:
-
Each equation contains variables on the left side of the equal sign, and constants on the right side of the equal sign.
-
Each equation does not limit the number of variables and the sequence of variables, but each variable in each equation can only appear once.
-
The command rule for variables is a lowercase string (x y xx yy abc is a valid variable name)
-
The coefficient of a variable is limited to integers and floating-point numbers. the format of a floating-point number cannot be 1.4e8. the coefficient is adjacent to the variable, and the coefficient cannot be 0.
-
Equations and equations are separated by commas (,).
Those who have learned linear algebra must know that they only need to abstract the equations into matrices and solve them by linear algebra. Therefore, you only need to resolve the input equations to the right matrix and variable list, and the rest of the solving process can be handed over to tools related to linear algebra.
Analysis
Lexical parsing
Lex in ply is used for lexical parsing. There are a lot of lexical parsing theories, but lex is very intuitive to use. it is to use regular expressions to parse text strings into tokens one by one, the following code implements lexical parsing using lex.
From ply import lex # blank tab press enter these invisible symbols are ignored t_ignore = '\ t \ r' # When parsing errors directly throw the exception def t_error (t ): raise Exception ('error {} at line {}'. format (t. value [0], t. lineno) # record the row number to locate the error. def t_newline (t): r' \ n + 't. lexer. lineno + = len (t. value) # supports c ++-style \ annotation def t_ignore_COMMENT (t ): r' \/[^ \ n] * '# command rule def t_VARIABLE (t) of the variable ): r' [a-z] + 'return t # constant command rule def t_CONSTANT (t): r' \ d + (\. \ d + )? T. value = float (t. value) return t # The token of the symbol header supported in the input. of course, the t_PLUS = r' \ + 'method is also supported to define the plus sign as tokenliterals =' + -,; = 'tokens = ('variable', 'constant') if _ name _ = '_ main __': data = '''-x + 2.4y + z = 0; // this is a comment 9y-z + 7.2x =-1; y-z + x = 8''' lexer = lex. lex () lexer. input (data) while True: tok = lexer. token () if not tok: break print tok
Directly run the file to print the parsed token string, as shown below. for detailed usage documents, see ply.
LexToken(-,'-',2,5)LexToken(VARIABLE,'x',2,6)LexToken(+,'+',2,8)LexToken(CONSTANT,2.4,2,10)LexToken(VARIABLE,'y',2,13)LexToken(+,'+',2,15)LexToken(VARIABLE,'z',2,17)LexToken(=,'=',2,19)LexToken(CONSTANT,0.0,2,21)LexToken(;,';',2,22)
Syntax parsing
In ply, yacc is used for syntax analysis. Although complex lexical analysis can replace simple syntax analysis, it is not competent to parse complex lexical analysis similar to programming languages. Before using yacc, you need to understand the context-independent grammar. This part is too complicated. I only need to understand some simple concepts. if you are interested, you can take a look at the compilation principles for a deeper understanding.
Currently, there are two types of syntax analysis methods: bottom-up analysis and top-down analysis. The top-down splitting method is a method that starts from grammar symbols and pushes forward to a given sentence according to grammar rules, or constructs a syntax tree from the root of the tree, until the analysis method of each leaf is established. The representative algorithm is LL (1). The grammar parsing capability of this algorithm is not strong, the syntax definition requirements are relatively high, and mainstream compilers are not used. The bottom-up analysis method starts from a given input string and gradually performs contract reduction according to grammar rules until it reaches the start symbol of the grammar, or starts from the end of the grammar book step by step, analysis Method until it is reduced to the root node. The algorithms include SLR and LRLR. ply uses LRLR.
Therefore, we only need to define Grammar and statute actions. The following is the complete code.
#-*-Coding = utf8-*-from ply import (lex, yacc) # blank tab press enter to ignore these invisible symbols t_ignore = '\ t \ r' # directly throw an exception def t_error (t) when parsing an error ): raise Exception ('error {} at line {}'. format (t. value [0], t. lineno) # record the row number to locate the error. def t_newline (t): r' \ n + 't. lexer. lineno + = len (t. value) # supports c ++-style \ annotation def t_ignore_COMMENT (t ): r' \/[^ \ n] * '# command rule def t_VARIABLE (t) of the variable ): r' [a-z] + 'return t # constant command rule def t_CONSTANT (t): r' \ d + (\. \ d + )? T. value = float (t. value) return t # The token of the symbol header supported in the input. of course, the t_PLUS = r' \ + 'method is also supported to define the plus sign as tokenliterals =' + -,; = 'tokens = ('variable', 'constant') # top-level grammar. the p [1] corresponding to equations is a list, contains the variables and coefficients on the left of the equation and the constant def p_start (p): "start: equations" var_count, var_list = 0, [] for left, _ in p [1]: for con, var_name in left: if var_name in var_list: continue var_list.append (var_name) var_count + = 1 matrix = [0] * (var_count + 1) for _ in xrange (len (p [1])] for counter, eq in enumerate (p [1]): left, right = eq for con, var_name in left: matrix [counter] [var_list.index (var_name)] = con matrix [counter] [-1] =-right var_list.append (1) p [0] = matrix, var_list # syntax of the equations. each equation is separated by or. def p_equations (p): "" equations: equation ', 'equations | equation '; 'equations | equation "if len (p) = 2: p [0] = [p [1] else: p [0] = [p [1] + p [3] # Syntax def p_equation (p): "" equation: eq_left '= 'eq_right "" p [0] = (p [1], p [3]) # Syntax def p_eq_left (p) on the left of the equation ): "eq_left: var_unit eq_left |" if len (p) = 1: p [0] = [] else: p [0] = [p [1] + p [2] # six syntax examples: x, 5x, + x,-x, + 4x, -4y # The Normalization Form is a tuples, for example, (5, 'x') def p_var_unit (p): "var_unit: VARIABLE | constant variable | '+ 'variable |'-'variable | '+ 'constant VARIABLE |'-'constant VARIABLE "" len_p = len (p) if len_p = 2: p [0] = (1.0, p [1]) elif len_p = 3: if p [1] = '+ ': p [0] = (1.0, p [2]) elif p [1] = '-': p [0] = (-1.0, p [2]) else: p [0] = (p [1], p [2]) else: if p [1] = '+': p [0] = (p [2], p [3]) else: p [0] = (-p [2], p [3]) # constant corresponding to the right side of the equation. example: 1.2, + 1.2,-1.2def p_eq_right (p): "" eq_right: CONSTANT | '+' CONSTANT | '-' CONSTANT "if len (p) = 3: if p [1] = '-': p [0] =-p [2] else: p [0] = p [2] else: p [0] = p [1] if _ name _ = '_ main _': data = '''-x + 2.4y + z = 0; // this is a comment 9y-z + 7.2x =-1; y-z + x = 8 ''' lexer = lex. lex () parser = yacc. yacc (debug = True) lexer. lineno = 1 s = parser. parse (data) print s
Run the file directly. the output is as follows. then, you can use the linear algebra method to solve the values of various variables.
([[-1.0, 2.4, 1.0, -0.0], [7.2, 9.0, -1.0, 1.0], [1.0, 1.0, -1.0, -8.0]], ['x', 'y', 'z', 1])
Summary
Based on python's concise syntax, ply provides us with a powerful syntax analysis tool. for more complex examples, refer to examples. There is such an artifact, a rowing boat!
The above is the content of the python development compiler. For more information, see The PHP Chinese website (www.php1.cn )!