[Compiling principle learning] lexical analysis

Previously has not been able to complete the system of the compilation principle, only a very superficial understanding, although in fact, the task in the work has no impact, but always feel missing a chunk of knowledge, coupled with the so-called programmer three Big Romantic (compiler, operating system, graphics) yearning, So recently with the introduction of the computer professional NetEase cloud class to learn the principles of compiling. Helpless nature lazy, often feel tired after work, playing games ah look at video Ah, procrastination is committed .... So here is going to learn the process, experience record down, also is a supervision of their own. Course Portal Http://mooc.study.163.com/learn/USTC-1000002001#/learn/announce

First of all to clear a concept, the language is designed out, not write code written out, the compiler is written out. The source code is actually a bunch of characters, but some characters are organized in accordance with the rules of pre-defined rule.

The compiler is a program that accomplishes the function of putting the source code (C/c++,python,java .... ) to the target code. This involves a lot of process. The simple part is the following process

Taking if (a > 1) as an example, the lexical Analyzer's task is to split this sentence into if, (, a,>,1,) and determine their type, such as if is a keyword, a is a variable, the name a,> is a greater than sign, 1 is an integer, a value of 1 and so on ...

Here is an example of a simple parser code that completes the following functions:

Parser input: A sequence of characters stored in a text file, derived from the ASCII character set. The file may contain four tokens: the keyword if, the C language compliant identifier, the space character, and the carriage return \ n.

Output of the parser: prints out the type of identifier identified, the line number, and the column number information.

1 #ifndef Tokenizer_h2 #defineTokenizer_h3 4#include <iostream>5#include <string>6#include <vector>7 using namespacestd;8 9 enumsymbolTen { One     //keyword If, C language-compliant identifier, space character, carriage return A KEYWORD, - ID, - BLANK, the ENTER - }; -  - structresult + { -     stringToken//symbol name +     intType//symbol Type A     intLine//line where the symbol is located at     intRow//The column where the symbol is - }; -  - classTokenizer - { -  Public: inTokenizer (void); -~tokenizer (void); to  +  Public: -     // the     voidReadsourcefile (Const string&fileName); *     // $     voidParse ();Panax Notoginseng     // -     voidHandletoken (Const stringTokenintLineintrow); the     // +     voidShowresult (); A Private: the     Char*M_buffer; +Vector<result>M_resultvec; - }; $  $ #endif

#include"Tokenizer.h"#include<fstream>#defineBuffsize 1024*1024Tokenizer::tokenizer (void) {M_buffer= (Char*) malloc (buffsize); memset (M_buffer,0, buffsize);} Tokenizer::~tokenizer (void){}voidTokenizer::readsourcefile (Const string&fileName) {FILE* fp = fopen (Filename.c_str (),"R"); if(FP! =NULL) {fread (M_buffer,1, BUFFSIZE,FP);    Fclose (FP); }    Else{cout<<"failed to open file"; }}voidTokenizer::P arse () {std::stringtoken; intCurrentLine =1;//mark the current number of rows    intpos =1;//Mark Token in the position of the line    inti =0; Charc =M_buffer[i];  while(c! =NULL) {        if(c! =' '&& C! ='\ n') {token.append (1, c); }        if(c = =' ') {Handletoken (token,currentline,pos);                    Token.clear (); POS= (i +1) +1;//The subscript for the next character is i+1. Represents column (I+1) +1.        }        if(c = ='\ n') {Handletoken (token,currentline,pos);            Token.clear (); POS=1; CurrentLine+=1; } C= m_buffer[++i]; }    //processing the last tokenHandletoken (Token,currentline,pos);}voidTokenizer::handletoken (Const stringTokenintLineintrow) {    if(token = ="")    {        return;    } result ret; Ret.token=token; Ret.line=Line ; Ret.row=Row; if(token = ="if") {Ret.type=KEYWORD; }    Else if(token = =" ") {Ret.type=BLANK; }    Else if(token = ="\ n") {Ret.type=ENTER; }    Else{Ret.type=ID; } m_resultvec.push_back (ret);}voidTokenizer::showresult () { for(inti =0; I<m_resultvec.size (); i++)    {        Switch(m_resultvec[i].type) { CaseKEYWORD: {cout<<m_resultVec[i].token<<"  "<<"("<<m_resultVec[i].line<<","<<m_resultVec[i].row<<")"<<Endl;  Break; }         CaseID: {cout<<"ID ("<<m_resultVec[i].token<<")"<<"  "<<"("<<m_resultVec[i].line<<","<<m_resultVec[i].row<<")"<<Endl;  Break; }         CaseBLANK: { Break; }         CaseENTER: { Break; }        }    }}

1#include"Tokenizer.h"2 3 intMain ()4 {5 Tokenizer Tokenizer;6Tokenizer. Readsourcefile ("./test.txt");7 Tokenizer. Parse ();8 Tokenizer. Showresult ();9 TenSystem"Pause"); One      A     return 0; -}

[Compiling principle learning] lexical analysis