Self-developed Compiler (6) Context-independent language and grammar

Last time, we have learned the principle and tool of lexical analysis, the first phase of syntax analysis, and introduced tools such as regular expressions, regular language, and DFA. This time, we will begin the most important phase involving the compiler frontend-syntax analysis. Simply put, this step requires a complete analysis of the syntax structure of the entire programming language. The last result of lexical analysis is to break down input strings into word streams, that is, words with specific meanings such as keywords and identifiers. A complete programming language, on which syntax rules of various declarations, statements, and expressions must be defined. Observe the programming languages we are familiar with, most of which have a recursive nature. For example, the four arithmetic operations and the expressions in parentheses, both sides of each operator can be any expressions. For example, 1 + a is an expression, (1 + a) * (2-c) is also an expression, (a + B) + c) * (d-e) is also an expression. For example, for the if statement, the if block and the else block can also be nested with the if statement. The Regular Expression and regular language introduced in lexical analysis cannot describe this structure. If we use DFA to explain it, DFA has only a limited state, and it cannot trace this infinite recursion. Therefore, the expressions in the programming language, andNot a regular language. We need to introduce a language with better performance --Context-independent language.

 

To introduce the context-independent language, Let's first look at the tool that defines the context-independent Grammar --Generate. We still use a programming language expression as an example, but this time we assume there are only three expressions-one representing the variable name identifier, the expressions enclosed in parentheses, and the addition of the two expressions. For example, a is a variable expression, a + B is the expression of adding two variable expressions, and (a + B) is a bracket expression. We use the symbol E to represent an expression. The three expressions can be defined:

E → id E → E + E E → (E)

This form of definition is calledGenerate. The symbol E on the left is calledNon-Terminator(Nonterminal symbol), Which indicates that the new symbol "grammar variable" can continue to be generated ". Symbol → it indicates that a non-terminator can be "generated. The blue id, +, and (and other symbols in the above formula are fixed words that no longer produce new thingsTerminator(Terminal symbol). Note that non-terminator can appear on the right of the generator, which is the source of recursive grammar. Production goes through a seriesDerivationTo generate various sentences completely composed of terminologies. For example, let's demonstrate the derivation process of the expression (a + B) + c:

E => E + E => (E) + E => (E + E) + E => (a + B) + E => (a + B) + c

=> Indicates replacing a non-terminator in the current sentence pattern with the content on the right of the generative expression. In the above derivation process, we expand the leftmost non-terminator in the sentence every time.Leftmost Derivation. Of courseRightmost Derivation, The difference is that the rightmost non-terminator in the sentence pattern is expanded every time:

E => E + c => (E) + c => (E + B) + c => (a + B) + c

It can be seen that the same result can have different derivation processes. When the leftmost derivation is used, the left side of the sentence gradually becomes only the terminator; while the rightmost derivation is just the opposite. During the derivation, the right side of the sentence gradually becomes only the Terminator, and the final result is the final result of the entire sentence becoming the Terminator. All sentences that conform to the syntax definition can be deduced using the syntax generation formula.

 

The purpose of syntax analysis is to parse the input word stream (a + B) + c to obtain itsSyntax analysis tree. Let's take a look at what the syntax analysis tree looks like. Take (a + B) + c as an example. The syntax analysis tree is like this:

Each node in the syntax analysis tree is a non-terminator or Terminator. The Terminator is a leaf node of the tree (without a subnode), rather than a subnode. Once we get the syntax analysis tree, we can easily perform subsequent semantic analysis. For example, the syntax of this expression is "add the variables represented by a and B first, then add the result to the variable represented by c ". Then how can we get the syntax analysis tree? In fact, in the just-generated derivation process, we can establish the syntax analysis tree by the way, as long as the non-Terminator is expanded, in the syntax analysis tree, add non-final Expansion results under the corresponding node to generate. The following uses an animation to demonstrate the process of generating the (a + B) + c syntax analysis tree through the leftmost derivation and rightmost derivation:

Leftmost Derivation	Rightmost Derivation

We can see that the leftmost derivation and rightmost derivation syntax analysis tree are the same, which proves that there are at least two different analysis methods for parsing the same input with the same syntax. Subsequent chaptersRecursive descentIt is a leftmost derivation analysis method, while another popular LR analyzer is based on rightmost derivation. Currently, the popular compiler development method is to construct a true syntax analysis tree in the syntax analysis phase, and then perform subsequent analysis by traversing the syntax tree, therefore, the process of leftmost derivation is not very different from that of rightmost derivation.

 

In the preceding example, expression (a + B) + c can only have one syntax analysis tree. However, some other syntax analysis inputs may have multiple syntax analysis trees.Ambiguity. The grammar just now is actually ambiguous (where? Please think about it), but in order to better express the danger of ambiguity, we will give a new example, which adds multiplication in the previous example:

E → id E → E + E E → E * E E → (E)

If the expression a * B + c is derived using the above formula, there are two possible leftmost derivation:

Leftmost derivation 1: E => E + E => E * E + E => a * B + c

Leftmost derivation 2: E => E * E => a * E + E => a * B + c

The two derived syntax trees are different:

Derivation 1	Derivation 2

As we discussed earlier, the syntax analysis tree will be used for semantic analysis in the next step. In semantic analysis, the differences between the preceding two syntax trees are mainly reflected in the priority of operators. If we use the syntax tree derived from 1, we should first multiply a and B and then add c. If we use the syntax tree derived from 2, we should first add B and c, then multiply by. Obviously, the computing results of these two semantics can be different. We do not want the same expression in programming languages to have two types of semantics. Therefore, syntax with ambiguity is not suitable for syntax analysis. In practice, a non-Ambiguous syntax should be used to ensure that the same program has only one syntax analysis tree. For example, we can modify the syntax formula above to enable the operator to have the feature of left combination, and give multiplication a priority higher than addition at the beginning:

F → id F → (E) T → T * F T → F E → E + T E → T

After the syntax is modified, expressions with a plus sign (+) are not allowed on both sides of the * sign, but only expressions with parentheses and variable names are allowed. At the same time, continuous Addition or multiplication must start from the left side. This limits the possible derivation methods. In the new method, expression a * B + c only has one syntax analysis tree:

The syntax we used last in miniSharp is very similar to this one. In practice, we usually need to carefully observe and think about whether the grammar used is ambiguous. If you have any questions about grammar, I suggest you refer to C # spec to define C # syntax in great detail, I believe that you will learn more about the syntax of a modern programming language after reading Spec. I will also introduce some common syntax structure design methods in the subsequent chapters.

 

At the end of this article, I would like to introduce a little more context-independent language. Some people may have been wondering from the very beginning why this language is "context-independent" with grammar? In fact, "context-free" means that all the statements in grammar can be unconditionally expanded to the content on the right of the arrow. In addition, there is a context-related syntax, which can be expanded only under certain conditions. Context-related languages are much more complex than context-independent grammar, and there is no common method to effectively parse context-related languages, so they are not used in programming language design. Students may have realized that even context-independent grammar and language are much more complex than regular expressions and regular language. I can't describe the nature of context-independent language in detail here, but I can give some tips to interested students. Just as regular expressions have an equivalent computing model -- finite automaton -- which can be used to parse a regular language, context-independent grammar also has an equivalent computing model --Push-down Automation(Put-Down Automation,PDA). In addition to a limited set of statuses and State transformations, the push-down machine also has an infinite capacityStack. Unlike the finite-state automation, the status of the push-down automation is not only transferred based on the input characters and the current status, but also based on the characters at the top of the stack; in addition, the push-down automatic mechanism must determine when to push or pop up characters to the stack. Similar to a Finite Automation, a push-down automation also exists.Uncertain push-down machine(NPDA) andDeterministic push-down machine(DPDA. The two kinds of push automatic machines areNot equivalent. The non-deterministic push-down mechanism corresponds to the entire context-independent language, while the deterministic push-down mechanism corresponds to a real subset of context-independent languages. NPDA has a much more powerful "Guess" capability than NFA, so that we cannot easily use computers to simulate it. We can only simulate DPDA for parsing. Fortunately, almost all programming languages use the syntax accepted by a DPDA. The syntax analysis mechanism we introduced in the next chapter, some of which cannot even reach the capabilities of DPDA, that is, we can only process a small part of context-independent grammar. But even this small part is enough to describe the C # syntax.

 

In the next article, we will introduce the implementation method of recursive descent syntax analyzer. Hope you continue to pay attention to my VBF project: https://github.com/Ninputer/VBF and my microblog: http://weibo.com/ninputer thank you for your support!