In-depth study of Clang (vii) Clang lexer code reading notes Lexer

Source: Internet
Author: User
Tags lexer



Sninning (snsn1984)



SOURCE Location: Clang/lib/lexer.cpp



Source Network Address: http://clang.llvm.org/doxygen/Lexer_8cpp_source.html






Lexer.cpp This file, is clang this front-end lexical Analyzer's main file, its content is the Lexer of this class, the specific implementation of the original file in the comments: "The" "This fileimplements the Lexer and Token Interfaces. " This explains the file, but token only implements two simple functions, and all that remains is the implementation of Lexer. So to figure out how Clang's lexical analyzer is implemented, you have to have a deep understanding of the file.



Start with the Lexer initialization function:





void Lexer::InitLexer(const char *BufStart, const char *BufPtr,
   56                       const char *BufEnd) {
   57   BufferStart = BufStart;
   58   BufferPtr = BufPtr;
   59   BufferEnd = BufEnd;
   60 
   61   assert(BufEnd[0] == 0 &&
   62          "We assume that the input buffer has a null character at the end"
   63          " to simplify lexing!");
   64 
   65   // Check whether we have a BOM in the beginning of the buffer. If yes - act
   66   // accordingly. Right now we support only UTF-8 with and without BOM, so, just
   67   // skip the UTF-8 BOM if it‘s present.
   68   if (BufferStart == BufferPtr) {
   69     // Determine the size of the BOM.
   70     StringRef Buf(BufferStart, BufferEnd - BufferStart);
   71     size_t BOMLength = llvm::StringSwitch<size_t>(Buf)
   72       .StartsWith("\xEF\xBB\xBF", 3) // UTF-8 BOM
   73       .Default(0);
   74 
   75     // Skip the BOM.
   76     BufferPtr += BOMLength;
   77   }
   78 
   79   Is_PragmaLexer = false;
   80   CurrentConflictMarkerState = CMK_None;
   81 
   82   // Start of the file is a start of line.
   83   IsAtStartOfLine = true;
   84   IsAtPhysicalStartOfLine = true;
   85 
   86   HasLeadingSpace = false;
   87   HasLeadingEmptyMacro = false;
   88 
   89   // We are not after parsing a #.
   90   ParsingPreprocessorDirective = false;
   91 
   92   // We are not after parsing #include.
   93   ParsingFilename = false;
   94 
   95   // We are not in raw mode.  Raw mode disables diagnostics and interpretation
   96   // of tokens (e.g. identifiers, thus disabling macro expansion).  It is used
   97   // to quickly lex the tokens of the buffer, e.g. when handling a "#if 0" block
   98   // or otherwise skipping over tokens.
   99   LexingRawMode = false;
  100 
  101   // Default to not keeping comments.
  102   ExtendedTokenMode = 0;
  103 }






In-depth study of Clang (vii) Clang lexer code reading notes Lexer


Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.