Go Language Black Magic

This is a creation in Article, where the information may have evolved or changed.

Today I'm going to teach you some useless skills, or you can call it artifice or black magic. Good use can improve performance, the use of bad will attract demons, hey.

Introduction to Black Magic

In order to let everyone learn the basic black magic can be realized, when necessary to create a part of this article to teach the magic of their own, here need to first lay a good foundation for everyone.

Before you learn the black magic of Go, you need to see the nature of the go world before you can acquire the same abilities as Neo.

What is the nature of slice in the go language? is a reflect. The memory that is pointed to by the data field in the Sliceheader struct and the struct. What is the nature of string? is a reflect. Stringheader the structure and the memory that the struct points to.

What is the nature of pointers in the go language? It's unsafe. Pointer and UIntPtr.

When you know the nature of them, you can play with them at will, hey.

First-get slice and string of memory data

Let me try, you have a CGO interface to invoke, you need to pass a string data or byte array data from the go side to the C side, like this: Mysql/conn.go at Master Funny/mysql GitHub

A variety of tutorials and documentation are available that tell you to use c.gostring or c.gobytes to transform your data.

But what happens when you call these two functions? At this point go copies a copy of the data and then passes the address of the new data to C, because go does not want to take any risks.

Your C program just want to use this data one time, but also have to do a copy of the data, this is a performance fetish is how terrible a fact!

At this point we need a black magic to do not copy the data and to pass the pointer address to C.

//returns &S[0], which is not allowed in gofunc Stringpointer(s string) unsafe.Pointer {P := (*reflect.Stringheader)(unsafe.Pointer(&s))return unsafe.Pointer(P.Data)}//returns &B[0], which is not allowed in gofunc Bytepointer(b []byte) unsafe.Pointer {P := (*reflect.Sliceheader)(unsafe.Pointer(&b))return unsafe.Pointer(P.Data)}

The above is the first black magic, we go first to the go string pointer, which is essentially a *reflect. Stringheader, but go tells us it's a *string, We tell go it is also a unsafe.pointer,go say it is, so you get unsafe.pointer, and then you escaped the surveillance of go, secretly unsafe.pointer turned into a *reflect. Stringheader.

With the *reflect. Stringheader, you quickly take the data field point to the memory address, it is go to protect you do not want to see the secret, you put this address secretly told C, so C on a pleasant peek at the privacy of go.

Second type-turn []byte] into string

You must laugh, it's not easy to turn []byte into a string]. The type conversion syntax for the Go language beginner: []byte (str).

But do you know the price of this? Since we are free to play with Sliceheader and Stringheader, why can't we make a string to go? Does the inside of go do this?

Let's start with the experiment:

 Package LABS28Import "Testing"Import "unsafe"func test_bytestring(T *Testing.T) {var x = []byte("Hello world!")var y = *(*string)(unsafe.Pointer(&x))var Z = string(x)if y != Z {T.Fail()}}func Benchmark_normal(b *Testing.B) {var x = []byte("Hello world!") for I := 0; I < b.N; I ++ {_ = string(x)}}func benchmark_bytestring(b *Testing.B) {var x = []byte("Hello world!") for I := 0; I < b.N; I ++ {_ = *(*string)(unsafe.Pointer(&x))}}

This experiment proves that we can use []byte's data to make a string to go. Then two sets of benchmark are tested, respectively, to test the efficiency of the common type conversion and the forgery of string.

The results are as follows:

$ go test -bench="."PASSBenchmark_Normal    20000000            63.4 ns/opBenchmark_ByteString    2000000000           0.55 ns/opok      github.com/idada/go-labs/labs28 2.486s

Yo West, obviously go this time again for stability to do some copy data and such things! How can this make a performance fetish endure!

I have a []byte] on hand now, but I want to use StrConv. Atoi () turns it into an integer value that literally means that a copy of the data needs to happen once to turn it into a string, like this: Mysql/types.go at Master Funny/mysql GitHub, this can't be tolerated!

Moves

// convert b to string without copyfuncbyteString(b[]byte)string{return*(*string)(unsafe.Pointer(&b))}

We take []byte's pointer, this time go and tell you it is *byte not *string, you tell it to roll Duzi this is unsafe. Pointer,go This and honest, and then you very comfortable to turn *byte into *string, because you know reflect. Stringheader and Reflect.sliceheader The structure of the end of a field only, the memory of the two is to it, no need to take the data field, go directly.

So, the world finally peace, hehe.

The third type-the structure and the []byte Mutual transfer

One day, you want to put a simple structure into binary data to save, when you think of Encoding/gob and Encoding/json, do a performance test, you think the efficiency is not possible higher?

So you try again Encoding/binady, performance also can, but you still not satisfied. But where is the bottleneck? You realize that the most efficient way is to completely do not parse the data or produce data Ah!

What do you do? It's time to use this black magic:

type MyStruct struct {A intB int}var sizeofmystruct = int(unsafe.Sizeof(MyStruct{}))func mystructtobytes(s *MyStruct) []byte {var x reflect.Sliceheaderx.Len = sizeofmystructx.Cap = sizeofmystructx.Data = UIntPtr(unsafe.Pointer(s))return *(*[]byte)(unsafe.Pointer(&x))}func bytestomystruct(b []byte) *MyStruct {return (*MyStruct)(unsafe.Pointer((*reflect.Sliceheader)(unsafe.Pointer(&b)).Data,))}

This is a tortuous but familiar story. You made a sliceheader, want to point its data field to your struct, but go tells you no, you put the go on one side as usual and you get unsafe. Pointer, but this time go has not forget, it tells you that data is uintptr,unsafe. Pointer is not uintptr, your big foot kicks it off and shouts: unsafe. Pointer is uintptr, you don't take these concepts to fool me, go fart run away, now you Yimapingchuan come to the Exit function, go unexpectedly already where waiting for you! You came forward rinsed to play it far, smoothly sliceheader the hand to turn into a []byte.

After a while, you get a []byte, you know you need to turn it into mystruct to read the data. Go is not your opponent at this time, it has washed the buttocks in the function entrance waiting for you, your line of code to solve it.

Type four-optimize GC with CGO

You are already a go world Neo,go and Ben can't get you how. But one day the GC of Go suddenly convulsions, originally this goods is regardless of how the object uses, each GC is given to the census, resulting in a long system pause time.

But you are a performance fetish, you put a lot of data in memory for quick access, you want to kick the butt of go again, but you can not, after all, you still in the world of Go, you now have to wipe the bottom of it, you seem to see go to hide in the side to steal laughter.

You think you have CGO, you can easily apply C to go outside of the memory, go GC does not scan this part of memory.

You also thought you could use unsafe. Pointer the pointer of C to the go structure pointer. So a large number of resident memory objects were turned into the heihu of Go world in this way, and the go GC was relaxed.

But you still have a lot of slice, so you use C application memory to Sliceheader to construct their own slice, so your slice have turned into go world Heihu, go GC finally calm.

But the prosperity is always not long, one day the go world suddenly collapsed, leaving only a word:segmentation Fault. What's wrong with you?

After an overnight investigation, you find that your Heihu object is secretly engaged with other legitimate residents of Go world, and when go world thinks that a resident has died, recycle it with a GC, but your underground world thinks it is alive and continues to visit it.

So you took some time to cut off all the connections and the world was quiet for the time being.

But you are already very tired, this time you think of a sentence:

for inaction, there is no died