Atitit. Lexical Analysis Implementation Token Attilax Summary

Atitit. The realization of lexical analysis token Attilax Summary

1. Lexical Analysis (English: Lexical analyses) and Token 1

1.1. Scanner 2

2. The word stream must be recognized as reserved words, identifiers ( variables ), constants, operators (operators) and bounds of the five major classes 2

2.1. Operator (Operators)::: 3

2.2.4. interface characters: " ; " semicolon ," {} "curly braces, single quotes, double quotes 3

3. How do I perform lexical analysis ? 3

3.1. Using an FSM State machine ( automaton) 3

4. Lexical Analyzer Framework Selection 4

4.1. Parser 4

4.2. Lex 4

4.3. Flex Lexical Analyzer 4

4.4.1 ANTLR Introduction 5

4.5. ANTLR Korean style JAVACC 5

4.6. ANTLR Introduction 6

5. Reference 6

1. Lexical Analysis (English: Lexical analysis) with token

is the process of converting character sequences into Word (Token) sequences in computer Science . The procedure or function for lexical analysis is called the lyrics Analyzer (Lexical Analyzer, referred to as Lexer), also known as a Scanner (Scanner). The lexical parser is generally present as a function for the parser to invoke.

The word here is a string that is the smallest unit that forms the source code . The process of generating a word from an input character stream is calledtokenization, in which the lexical analyzer also classifies words.

Lexical analyzers generally do not care about the relationship between words (in the context of syntactic analysis), for example: the lexical analyzer can recognize parentheses as words, but does not guarantee that the parentheses match.

Lexical parsing (lexical analysis) or scanning (scanning) is the first step in a compiler. The lexical parser reads the stream of characters that make up the source program, organizes them into a sequence of meaningful morphemes (Lexeme), and produces lexical units (tokens) as output for each morpheme.

In simple terms, the lexical analysis is to read the source program (which can be considered a very long string) and "cut" into small segments (each section is a lexical unit token), each of which has a specific meaning, for example, to represent a particular keyword or to represent a number. And this lexical unit in the source program corresponding to the text, is called "morphemes."

Token is the word that makes the sentence of the program similar to the word segmentation.

Author:: Old Wow's paw attilax Ayron, email:[email protected]

Reprint please indicate source: Http://blog.csdn.net/attilax

1.1. Scanner

The first stage of lexical analysis is scanners, usually based on finite state automata . The scanner is able to identify all the sequences of characters that may be contained in the word it can handle (a single such sequence of characters, the " morpheme "mentioned earlier). For example , an " integer " Word can contain all numeric character sequences. In many cases, the type of the word can be pushed out according to the first non-whitespace character, so that the subsequent characters can be processed one after the other until a character that is not part of that type of word character set (that is , the longest consistent principle ) is present.

Although in some cases it is necessary to manually write a lexical parser, the lexical analyzer is typically generated using automated tools.

Htmltokenizer processing, it is the use of a poor state automaton to complete the lexical parsing, the decoded string as input, output one by one htmltoken.

To know that the htmltoken in fact is to save the input stream is divided into a few pieces of data, which will be used to build the DOM node. These data are abstracted as a record of types, data, attributes combined.

, the lexical analysis of HTML is realized by using the state machine in Htmltokenizer. HTML parsing is based on the type of Htmltoken, as well as the state machine in the Htmltreebuilder to identify, and then based on the type of recognition to create a specific node, and after the node is created, insert it into the DOM tree corresponding position, Complete the parsing and generate a DOM tree as a syntax tree.

2. The word stream must be recognized as a reserved word, identifier (variables), constants, operators (operators), and bounded characters in five major categories

2.1. Operator (Operators):::

() [].

? :

Conditions

From right to left

() [].

Parentheses (functions, etc.), arrays, two struct members access

From left to right

,	Comma (order)
+ -	Add, Subtract	From left to right

Brackets , spinning brackets

Reference

Compiler DIY--Lexical analysis -godlike- Blog channel -CSDN.NET.htm

2.2.4. Bounds: ";" semicolon, "{}" curly brace, single quote, double quotation mark 3.how to doLexical Analysis？

A: A very simple idea is to use a state to hold the state in the processing to each character, such as an identifier or a number or a space, and so on, until the state changes to can be identified as a different token Time to end.

To be sure, it is necessary to determine the data that needs to be processed in one character, then truncate it in the proper place and get a token.

The core here is to distinguish the characters corresponding to the different symbols, to truncate the symbol when a character cannot express it, and token to form.

3.1. Using an FSM state machine (automaton)

4.Lexical AnalyzerFrame Selection4.1.Parsing Parser4.2.Lex4.3.Flex Lexical Analyzer

Using a calculator For example,12+34*9 This section of the " source program " lexical analysis process is as follows:

Fig. 2 Lexical analysis process of calculation

A string that is not meaningful to a computer, and has been parsed to give a slightly meaningful Token stream. Digit means that the lexical unit corresponds to a number,theoperator is the operator, and the corresponding number and symbol (pink background) is the morphemes. At the same time, some unnecessary blanks and annotations in the program can be filtered out by the lexical analyzer, so that the subsequent steps of parsing are much easier to process.

Using ANTLR or JAVACC to generate a lexical analyzer is relatively straightforward, and it's a very troublesome thing to write yourself.

Open source ll (K) syntax/lexical analyzer-ANTLR

4.4.1 ANTLR Introduction

antlr-a, formerly known as Pccts, provides a recognizer (recognizer) that automatically constructs a custom language through syntax descriptions for languages including java,c++,c#, compiler (parser) and the Interpreter (translator) framework. ANTLR can resolve the recognition conflict by asserting (predicate), supporting actions and return values (return value), and, better, automatically generating a syntax tree from the input and visualizing it (which I'll demonstrate in the following example). As a result, the translation of computer language into a common task-before this yacc/lex appears to be too academic school, and LL (k)-based ANTLR although in the efficiency of the slightly insufficient, but after recent changes in the upgrade, so that antlr enough to cope with the vast majority of existing applications. Thanks to Dr. Terence Parr and his colleagues for their excellent work over the past more than 10 years, they have done a lot of basic work on the basis of compiling theory and the construction of language tools, and have directly led to the production of Russian ANTLR. nother Tool for Language Recognition

Regular Expressions       Regular Expressions    recognized       is the    preferred for text processing     Tools When we use regular table    show   ,   first    define    a regular expression,      Post    and    expected    text       match   ,   most    then follow the    regular table      -      in       Group   ,         get       matches          according to   ,      after       next    one    step    processing (   output   ,      ,   , etc.   ). Use regular Expressions,   overall when       more complex    some    problem       handling    over    turndown    longer  &nbsp, and sometimes    to    process an     problem   , write    out   Regular Expressions    obscure   ,   very    not    then       maintenance    .       Lexical analyzer       in      Antlr     morphemes Analyzer    using       and language    analyzers       and        Technology       structuring   ,   word      ,       Token       matching    use       recursion     Down    down       strategy  &nbsp, so       lexical analyzer    has processing    up    down    irrelevant    the     &nbsp of the    force;       Regular Expressions       able to process the    method    contains only    regular text        (   linear    text      ),   so    lexical analyzer       with &Nbsp;  processing    very    multi-regular expressions    difficult      -dealt with    issues    , such as    left       and    right,      ,    Matching      .       in addition   , in the      Antlr    morphemes Law Analyzer    want    match       lexical       #   ,  -Pass       Mutual citation       Square   -  -  -nested    and    recursive    definition   , than regular expression    book    write    more intuitive   ,    More       maintenance   .       Total       for   , using      ANTLR     Lexical Analyzer    processing text and regular expressions    phase    ratio, processing can    force more    powerful,       in development and    test    In the &nbs of this articlep;  Follow-up    department    Sub-   we're together    to    see    one       How to use      Antlr    Lexical analyzer    complete extraction, conversion, rewriting    this    three    class text processing work

4.5. ANTLR Korean style JAVACC

ANTLR Baidu for you to find relevant results of about 1,640,000

Is twice times the JAVACC ... So, Korean style antlr ...

Use two ANTLR keywords in the first line of the file Lexer Grammar declaration This is a lexical file, such as

The only thing to note is that the lexical name must be the same as the file name, or ANTLR will generate an error when generating the parser, which is similar to Sqlextrator . g contains grammar xxx; Names must be identical, where sqlextrator is used uniformly.

4.6. ANTLR Introduction

1. antlr  a tool for language recognition   (another tool for language recognition )   JAVA, C++,  C#  Action ( action 40  The development of the year is quite mature, using   ANTLR &NBSP,

2. There are many types of language recognition tools, such as the famous Lex and YACC, which have their open source version, Flex and Bison. In the Java Community, in addition to ANTLR , the language recognition tool also has JavaCC and SABLECC and so on.

An arithmetic expression is used to4class notation( in theANTLRis called theToken), which are identifiers, respectivelyID, which represents a variable; constantINT, which represents a constant; line breakNEWLINEand SpacesWS, the empty characters will be skipped when the language is processed,Skip ()is a method of the lexical parser class. such as the list3as shown:

4.6.1.1.1. Listing 3. notation definition

ID: (' a ' ... ') Z ' | ' A '.. '  Z ') +; INT: ' 0 ' ... '  9 ' +; NEWLINE: ' \ r '?  ' \ n '; WS: (' | ') \ t ' | ' \ n ' | ' \ r ') + {skip ();};

5. Reference

Open Source Parser--ANTLR-Shide's Column-Blog channel-CSDN.NET.htm

Lexical analysis - Wikipedia , the free encyclopedia . htm

( detailed ) Lexical analysis (string analysis) -Thinker-blogjava.htm

Browser exploration--webkit section--parsing HTML(3)htmltoken processing -  A column of Fallen souls -  Blog Channel -CSDN.NET.htm

C # Lexical Analyzer (i) lexical analysis introduces update 2014.1.8-CYJB- Blog Park . htm

Lexical Analysis (NFA and DFA)-woaidongmao-c++ blog . htm

Java Open Source Syntax Analysis generator category List . htm

( detailed ) Use ANTLR Working with text _ Baidu Library . htm

(IBM Details ) Use ANTLR Development Domain Language . htm

Lexical analysis, so that the state machine rotation more violent it ---- small C language - Chen's share - Blog Channel -CSDN.NET.htm

Lexical Analysis (Java Implementation ) without state machine -0≡(-∞, + ∞)- Blog channel -CSDN.NET.htm

Atitit. Lexical Analysis Implementation Token Attilax Summary