Go 1.8RC3 Source Code learning: Token

This is a creation in Article, where the information may have evolved or changed.

Objective

The token package contains data structures and methods related to Golang lexical analysis, and the source code is located in <go-src>/src/go/token

Token.go

The comments in the source code are great!

Token type

Token is the set of lexical tokens of the Go programming language

type Token int

Tokens

The list of tokens (token IDs)

const (    // Special tokens    ILLEGAL Token = iota    EOF    COMMENT    literal_begin    ...    literal_end    operator_beg    ...    operator_end    keyword_beg    ...    keyword_end)

Using the const definition of the Go language tokens, here's a place worth learning: using Xxx_beg and xxx_end as a different token group boundary, it's convenient to quickly determine the token type

Next is the token string description (token string) corresponding to the above const one by one

var tokens = [...]string {    ILLEGAL: "ILLEGAL",    EOF:     "EOF",    COMMENT: "COMMENT",    ...}

Query token string based on token ID

Queries the tokens array before checking the array out of bounds

func (tok Token) String() string {    s := ""    if 0 <= tok && tok < Token(len(tokens)) {        s = tokens[tok]    }    if s == "" {        s = "token(" + strconv.Itoa(int(tok)) + ")"    }    return s}

Keywords

var keywords map[string]Token

Summarize