Sninning (snsn1984)
SOURCE Location: Clang/lib/lexer.cpp
Source Network Address: http://clang.llvm.org/doxygen/Lexer_8cpp_source.html
Lexer.cpp This file, is clang this front-end lexical Analyzer's main file, its content is the Lexer of this class, the specific implementation of the original file in the comments: "The" "This fileimplements the Lexer and Token Interfaces. " This explains the file, but token only implements two simple functions, and all that remains is the implementation of Lexer. So to figure out how Clang's lexical analyzer is implemented, you have to have a deep understanding of the file.
Start with the Lexer initialization function:
void Lexer::InitLexer(const char *BufStart, const char *BufPtr,
56 const char *BufEnd) {
57 BufferStart = BufStart;
58 BufferPtr = BufPtr;
59 BufferEnd = BufEnd;
60
61 assert(BufEnd[0] == 0 &&
62 "We assume that the input buffer has a null character at the end"
63 " to simplify lexing!");
64
65 // Check whether we have a BOM in the beginning of the buffer. If yes - act
66 // accordingly. Right now we support only UTF-8 with and without BOM, so, just
67 // skip the UTF-8 BOM if it‘s present.
68 if (BufferStart == BufferPtr) {
69 // Determine the size of the BOM.
70 StringRef Buf(BufferStart, BufferEnd - BufferStart);
71 size_t BOMLength = llvm::StringSwitch<size_t>(Buf)
72 .StartsWith("\xEF\xBB\xBF", 3) // UTF-8 BOM
73 .Default(0);
74
75 // Skip the BOM.
76 BufferPtr += BOMLength;
77 }
78
79 Is_PragmaLexer = false;
80 CurrentConflictMarkerState = CMK_None;
81
82 // Start of the file is a start of line.
83 IsAtStartOfLine = true;
84 IsAtPhysicalStartOfLine = true;
85
86 HasLeadingSpace = false;
87 HasLeadingEmptyMacro = false;
88
89 // We are not after parsing a #.
90 ParsingPreprocessorDirective = false;
91
92 // We are not after parsing #include.
93 ParsingFilename = false;
94
95 // We are not in raw mode. Raw mode disables diagnostics and interpretation
96 // of tokens (e.g. identifiers, thus disabling macro expansion). It is used
97 // to quickly lex the tokens of the buffer, e.g. when handling a "#if 0" block
98 // or otherwise skipping over tokens.
99 LexingRawMode = false;
100
101 // Default to not keeping comments.
102 ExtendedTokenMode = 0;
103 }
In-depth study of Clang (vii) Clang lexer code reading notes Lexer