Golang upgrade to 1.7, before the correct function error, analysis of the causes and solutions

This is a creation in Article, where the information may have evolved or changed.

After recently trying to upgrade the development environment to Golang1.7.1, the program will occasionally be down, after viewing the log, it is always found in a computed slice of the hash value of the place, the error message is:

Unexpected fault address 0xc043df4000,fatal Error:fault

Before 1.7, the program continued to run for 2 years, and never had this problem, suspected to be caused by the Golang compiler upgrading to SSAS. Refine your program's code to the following function:

//The main function of this code is to add up the value of a string's assii. func SIMPLECRC(ptr UIntPtr, size int) int {  ret := 0  maxptr := ptr + UIntPtr(size)   for ptr < maxptr {    b := *(*byte)(unsafe.Pointer(ptr)) //Where the error occurred    ret += int(b)    ptr++  }  return ret}

Note: The actual code is much more complex than this. With this kind of writing, the performance is up to 8 times times higher than the conventional notation.

Parsing errors directly result from "Illegal memory address access", only one reason is "the memory used by the string was freed by SSA compilation", was recycled by GC and returned to
Windows operating system. Therefore, the principle of the SSA compiler is consulted. The SSA compiler is found to be much smarter, and it can be quickly judged based on (established rules) that memory is no longer being used, so memory recovery is very rapid. The focus of this thinking becomes: Is there any way to tell the SSA compiler that specific memory is not recycled in the specified code area? , remember to see Golang1.7 in the runtime package, add a function func KeepAlive (interface{}) {}, after viewing the note, "Use this function to set the memory to remain valid in the specified code area" without being recycled by GC.

To reproduce the above inference, write the following example:

//Memtest Package MainImport (    "FMT"    "Reflect"    "Runtime"    "unsafe")func SIMPLECRC(ptr UIntPtr, size int) int {    ret := 0    maxptr := ptr + UIntPtr(size)     for ptr < maxptr {        b := *(*byte)(unsafe.Pointer(ptr))        ret += int(b)        ptr++    }    return ret}//Simulate request memory, trigger GC Reclaim memoryfunc Allocation(size int) {    var  Free []byte     Free =  Make([]byte, size)    if Len( Free) == 0 {        Panic("Allocation Error")    }}func slicecrctest(Slice []byte, N int) (ret int) {    Newslice := []byte(string(Slice))                       //Get Independent memory    SH := (*reflect.Sliceheader)(unsafe.Pointer(&Newslice)) //Reflective slicing structure    ptr, size := UIntPtr(SH.Data), SH.Len                   //Get address size    Runtime.GC()                                            //Force memory Reclamation     for I := 0; I < N; I++ {        ret = SIMPLECRC(ptr, size) //Calculate CRC Check Code        Allocation(size)           //Simulate request memory, trigger GC Reclaim memory    }    //runtime. KeepAlive (Newslice)//Once the Bank has commented, the result is no longer 1665, the uncomment section is correct    return}func stringcrctest(Str string, N int) (ret int) {    Newstr := string([]byte(Str))                          //Get Independent memory    Runtime.Setfinalizer(&Newstr, func(x *string) {})      //Set up recycling events    SH := (*reflect.Stringheader)(unsafe.Pointer(&Newstr)) //Reflection string structure    ptr, size := UIntPtr(SH.Data), SH.Len                  //Get address size    Runtime.GC()                                           //Force memory Reclamation     for I := 0; I < N; I++ {        ret = SIMPLECRC(ptr, size) //Calculate CRC Check Code        Allocation(size)           //Simulate request memory, trigger GC Reclaim memory    }    //runtime. KeepAlive (NEWSTR)//Once the Bank has commented, the result is no longer 1665, the uncomment section is correct    return}func Main() {    var B = []byte("1234567890-1234567890-1234567890") the value of the//CRC is: 1665    var S = string(B)                                  //Generate string    N := 1000000                                       //Loop execution 1,000,000 times    FMT.Printf("SIMPLECRC (\"%s\ ") =%v\n", B, slicecrctest(B, N))    FMT.Printf("SIMPLECRC (\"%s\ ") =%v\n", B, stringcrctest(S, N))}

The idea of reproducing the above code is to first request memory, specifically a new slice or string (whose value is "1234567890-1234567890-1234567890", and its correct CRC result is 1665), Pass in functions Slicecrctest and stringcrctest respectively to see the results of the operation, here only slicecrctest function of the internal implementation of ideas, stringcrctest and slicecrctest very consistent, please analyze their own understanding.

Inside the Slicecrctest function, the first is the code

Newslice: = []byte(string(slice))                       //Get Independent memory

Our code repeats two memory requests, which is designed to produce a local variable that accelerates the recurrence of GC recycling newslice.

SH: = (*reflect. Sliceheader) (unsafe//Reflective sectioning structure

The line of code is to obtain the data structure of the slice newslice by reflection, in order to read the first address and length of "1234567890-1234567890-1234567890".

PTR, size: = UIntPtr (Sh. Data), Sh. Len                   //Get address size

Our code is to get the first address and length of "1234567890-1234567890-1234567890" to the variable ptr, size.

Runtime. GC ()                                           //Forced memory reclamation

Our code is to force a memory recycle scan and then for loop 1 million times, so the purpose is to allow enough time for GC to reclaim memory, and the loop body class executes code as follows.

//Calculate CRC Check Code Allocation (size)           //Simulate request memory, trigger GC Reclaim memory

Call SIMPLECRC to calculate the checksum of "1234567890-1234567890-1234567890" and save the last result to the RET return variable (the correct value is 1665). The allocation function simulates applying a memory once, and the memory is reclaimed by the GC after the function returns.

//runtime. KeepAlive (newslice)//Once the bank has commented, the result is no longer 1665, the uncomment section is correct

This statement is most critical, this statement is annotated, then the result of Slicecrctest should be 0, which means that the newslice memory is recycled by GC, and the same piece of memory is again assigned to the free variable in the allocation function, because the initialization of free is 32 The ' 0 ' consists of slices, so the slicecrctest calculation turns into "0". The problem recurs, and the SSA compiler mistakenly thinks that memory is not valid, so the GC is recycled.

Note: In the actual reproduction process, because this is a random process, the different operating systems may not reproduce, but as long as the idea and principle, a slight adjustment of the value of N, the increase will be reproduced.

1000000                                       //Loop execution 1,000,000 times

Summary:

Due to the Golang of the SSA compiler, it becomes very smart and therefore uses reflection reflect. Stringheader,reflect. The block of memory pointed to by the uintptr in the Sliceheader return value is recycled as a block of memory that has not been used.

There are two solutions:

1. One is to try not to over-pursue performance, using reflection reflect and unsafe package functions. This avoids some weird, hard-to-analyze bugs.
2. If you want to use functions within the reflection reflect and unsafe packages, be aware that you must use runtime. KeepAlive tells the SSA compiler not to reclaim blocks of memory within the specified code snippet.