Modify the go language (golang) compiler source code to make it support UTF-8 BOM

The first official version of Go language (golang) go1 was released, but this emerging programming language is still very imperfect. No, I (liigo) found that its compiler does not support compiling the. Go source file with BOM UTF-8 encoding. This is strange, the language clearly requires that the source code file. Go must be a UTF-8 code, but there is a UTF-8 BOM not allowed. You know, there are too many BOM files in this world, many text editors/code Editors/ide will generate UTF-8 files with BOM by default. If the source code file has more Bom, the compiler will not be able to compile the file. I think it is too low.

Go language compiler (GC) does not support UTF-8 source files with BOM:
Golang 'S Compiler (GC) Don't accept the. Go files with UTF-8 BOM:

E:\liigo\golang\src>go run hello.gopackage :hello.go:1:1: illegal character U+FEFFE:\liigo\golang\src>go run hello.go# command-line-arguments.\hello.go:9: illegal UTF-8 sequencece d2

Fortunately, the Go language is an open-source project, and I (liigo) contribute code to support Compilation of. Go source code files with UTF-8 Bom. After analysis, it is found that the go language compiler (GC) Source Code involves reading from disk files. go file: a lexical analyzer (src/CMD/GC/lex) written in C language. c). One is written in the go language. go file parser (src/PKG/go/parser/interface. go ). The solution is also very simple, that is, read the file content from the disk, judge the first three bytes, and UTF-8 BOM three bytes (0xef, 0xbb, 0xbf) check, if they are consistent, the three bytes are ignored. The fourth byte is counted as the real content of the file, and then handed over to the lexical analyzer and parser for processing. Everything will work normally in the future.

The Lexical analyzer of Go language is manually written in C language. The lib9/libbio library is used as a disk file read/write buffer, it can "anti-read" the last four bytes. That is to say, if I just read ABCD as a byte, now I "anti-read" the last two bytes, in fact, it is equivalent that I just read AB and haven't read CD. With this "where read" mechanism, it is easy to ignore the UTF-8 file at the beginning of the BOM: First read the first three bytes, if the three bytes are exactly the three bytes of the UTF-8 BOM (0xef, 0xbb, 0xbf), then the three bytes that have just been read will be thrown away, the subsequent lexical analyzer reads from the bytes after the BOM exactly; if the three bytes that have been read are not UTF-8
For Bom, We need to "reverse read", that is, put them back, as I have never read them. The modified code is as follows:

// src/cmd/gc/lex.c : 

  319                 // Try to read and ignore UTF-8 BOM  320                 c1 = Bgetc(curio.bin);  321                 c2 = Bgetc(curio.bin);  322                 c3 = Bgetc(curio.bin);  323                 if(c1 != 0xef || c2 != 0xbb || c3 != 0xbf) {  324                         // If not UTF-8 BOM, restore the bytes.  325                         // Bungetsize > 3, so we can safely call Bungetc() 3 times.  326                         Bungetc(curio.bin);  327                         Bungetc(curio.bin);  328                         Bungetc(curio.bin);  329                 }

The Go source code Parser (PKG/go/parser) is compiled by the go language. Its function is to parse the. Go source code file as the syntax tree. Go build commands officially provided by go use PKG/go/parser to analyze and process library dependencies before compilation. The Go build command (PKG/go/parser) reads the. Go file into the memory and then parses it. The job I want to do is to delete the existing UTF-8 BOM before PKG/go/parser starts the formal parsing, this work only involves the basic operations of byte slice in the go language and is a lightweight and inexpensive operation.

// src/pkg/go/parser/interface.go : 

+// The data read from .go files maybe start with the UTF-8 BOM(byte order mark),+// we ignore the bytes here to make sure that the parser parses properly.+//+func ignoreUTF8BOM(data []byte) []byte {+if data == nil {+return nil+}+if len(data) >= 3 && data[0] == 0xef && data[1] == 0xbb && data[2] == 0xbf {+return data[3:]+}+return data+}+ // If src != nil, readSource converts src to a []byte if possible; // otherwise it returns an error. If src == nil, readSource returns // the result of reading the file specified by filename.@@ -27,22 +40,23 @@ case string: return []byte(s), nil case []byte:-return s, nil+return ignoreUTF8BOM(s), nil case *bytes.Buffer: // is io.Reader, but src is already available in []byte form if s != nil {-return s.Bytes(), nil+return ignoreUTF8BOM(s.Bytes()), nil } case io.Reader: var buf bytes.Buffer if _, err := io.Copy(&buf, s); err != nil { return nil, err }-return buf.Bytes(), nil+return ignoreUTF8BOM(buf.Bytes()), nil } return nil, errors.New("invalid source") }-return ioutil.ReadFile(filename)+fileData, err := ioutil.ReadFile(filename)+return ignoreUTF8BOM(fileData), err }

I submitted the modified Code to the Go language official source code library. The address on the code review page is:
Http://codereview.appspot.com/6036054/, or http://codereview.appsp0t.com/6036054 /.

However, the author/official developer of the Go language rejects this improvement. Rob Pike, the boss of the Go Development Team, personally replied and gave the reason for rejection:

Strictly speaking, a BOM is legal in UTF-8 but only as a marker forthe type of the data stream, a magic number if you will. Since Gosource code is required to be UTF-8, a BOM is never necessary andarguably erroneous. We've come this far without accepting BOMS and I'dlike to keep it that way.

In my opinion, the reason is very stubborn, and it is not even true logically. Oh, since it is stipulated that. Go must use UTF-8 encoding, so must not add UTF-8 Bom? After adding Bom, I refused to accept it? As mentioned above, many text editors will automatically add UTF-8 Bom, how to here is not legal. A very realistic example is that the portable program (notepad.exe) in Windows XP/Windows 7 systems will automatically add the UTF-8 BOM when saving the UTF-8 file. That is to say, the. Go source code you want to save in Notepad cannot be compiled. But it is such a serious problem that the go authors are not the same. The problems that can be improved so easily are that they refuse to improve. It creates a new resistance for users to use the go language. We can still understand the compromise of the technical solution. However, we have to stick to our opinions in irrelevant places rather than consider convenience for users. I can only say that they are dead-headed. I have encountered more than once in a similar situation. The summary is as follows:Technical experts are handsome and do harm to their productsThe cost of building a car behind closed doors is that you don't know how to die. (This summary also sounded the alarm for us to make easy-language products: we absolutely cannot simply make products with the mentality of technicians .)

Bytes ---------------------------------------------------------------------------------------------------

LiigoNote:

Golang 1.1 released in May 13, 2013 finally supports source code files with UTF-8 BOM:

"The Unicode byte order mark U+FEFF, encoded in UTF-8, is now permitted as the first character of a Go source file."

Http://golang.org/doc/go1.1#unicode

This day, the Go language development team boss Rob Pike refused to UTF-8 BOM that time (, content see the above), more than a year has passed, maybe he has forgotten what he said.

In addition, their idea of modifying source code is also the same as that of me (both modified Lex. c In the same place; although I did not directly modify the parser, but also indirectly achieved the goal by modifying the parser ):

Cc2bca9c03ef
Rob Pike, 2012-9-10: GC: initial BOM is legal

3d58333e8e2a
Russ Cox, 2012-10-7: CMD/GC: Skip over reported BOMs

4105c8cdc599 by Robert
Griesemer, 2012-9-7: Go/example: skip first character if it's a BOM

30444b809a9e by Robert
Griesemer, 2013-4-12: Go/clusters: reject BOMs that are not at the beginning

No! The three authors (including the two bosses) submitted at least four modifications, with a span of seven months before and after, finally achieved the effect of one-time modification and source code submission.