Use the Go language to bypass page cache read and write files

Sometimes we want our data to be dropped directly from the page cache buffer. In the go language, this can be achieved by opening a file with the parameter direct.

However, there is a hard requirement, that is, when reading and writing, the corresponding data in memory address must meet 512 alignment, that is, the first address of the 2 binary form at least 9 0 end, and the data length of 512 bytes of integer times, or read and write will fail.

We use the slice slice in the go language to verify the event.

First we create a go language slice and arbitrarily assign some values:

BUF: = Make ([]byte, 8192) for I: = 0; I < 20; i++ {Buf[i] = byte (i)}

Let's first try the normal read and write file process, bypassing the page cache without using the direct parameter first.

Func writewithoutalignmentwithoutdirect (buf []byte) {//Open filefile, err: = OS. OpenFile ("/dev/sdb", OS. O_wronly|os. O_create, 0666) if err! = Nil {fmt. Printf ("An error occurred with file opening or creation\n") Return}defer file. Close ()//write Filefmt. Println ("Buffer", unsafe.) Pointer (&BUF)) fmt. Println ("buffer[0]", unsafe. Pointer (&buf[0])) Buf2: = Buf[4:516]fmt. Println ("Write with buffer", unsafe. Pointer (&buf2[0]) _, err = file. Writeat (buf2, MB) If err! = Nil {fmt. Println ("Write error", err)} else {fmt. Println ("Write Succeed")}}

The result of this code operation is as follows:

Buffer  0xc42000a2a0buffer[0]  0xc42005a000write with buffer  0xc42005a004write succeed

It can be seen that the address of this slice is 0xc42000a2a0, the first address of the data in this slice is 0xc42005a000, which is a different from the C language, the first address of the C language is the first address of the data, and in the go language, the address of the slice and the first address of the data in the slice is different.

The first address of the data we want to write is from the 5th element of the slice, with the first address 0xc42005a004, although not 512 aligned, but since we did not attempt to bypass page cache, the return value of the Writeat function, err, can be seen, Our write operation was successful.

Let's try opening the file with the direct parameter, bypassing the page cache for the write data operation, and the other parameters unchanged.

Func writewithoutalignmentwithdirect (buf []byte) {//Open filefile, err: = OS. OpenFile ("/DEV/SDC", OS. O_wronly|os. O_create|syscall. O_direct, 0666) if err! = Nil {fmt. Printf ("An error occurred with file opening or creation\n") Return}defer file. Close ()//write Filefmt. Println ("Buffer", unsafe.) Pointer (&BUF)) fmt. Println ("buffer[0]", unsafe. Pointer (&buf[0])) Buf2: = Buf[4:516]fmt. Println ("Write with buffer", unsafe. Pointer (&buf2[0]) _, err = file. Writeat (buf2, MB) If err! = Nil {fmt. Println ("Write error", err)} else {fmt. Println ("Write Succeed")}}

After this code is run, we can see the following results.

Buffer  0xc42000a2e0buffer[0]  0xc42005a000write with buffer  0xc42005a004write error  WRITE/DEV/SDC: Invalid argument

See the Write Error,writeat function this time the return value given is not nil, our write operation failed, its return value returned an incomprehensible invalid argument (illegal parameter).

But we have no problem with the parameters Ah! Let's try to change the data you want to write to 512-aligned.

Func writewithalignmentwithdirect (buf []byte) {//Open filefile, err: = OS. OpenFile ("/DEV/SDD", OS. O_wronly|os. O_create|syscall. O_direct, 0666) if err! = Nil {fmt. Printf ("An error occurred with file opening or creation\n") Return}defer file. Close ()//write Filefmt. Println ("Buffer", unsafe.) Pointer (&BUF)) fmt. Println ("buffer[0]", unsafe. Pointer (&buf[0])) Buf2: = Buf[512:512+512]fmt. Println ("Write with buffer", unsafe. Pointer (&buf2[0]) _, err = file. Writeat (buf2, MB) If err! = Nil {fmt. Println ("Write error", err)} else {fmt. Println ("Write Succeed")}}

After this code runs, the results are as follows.

White with alignment and Direct:buffer  0xc42000a340buffer[0]  0xc42005a000write with buffer  0xc42005a200write succeed

Our write operation was successful! There is only one difference between this code and the last unsuccessful, that is, the first address to write the data is changed to 512 alignment.

Through these three go programs, we have clearly verified the conditions that bypass the page cache to write files.

Similarly, the following is a code that verifies that a 512-aligned condition is required to bypass the page cache read file.

Func readwithoutalignmentwithoutdirect (buf []byte) {//Read filefile, err: = OS. OpenFile ("/dev/sdb", OS. O_rdonly, 0666) if err! = Nil {fmt. Printf ("An error occurred whit file ipening.\n") Return}defer file. Close () buf = buf[2:514]fmt. Println ("Read with buffer", unsafe. Pointer (&buf[0]) _, err = file. ReadAt (buf, MB) If err! = Nil {fmt. PRINTLN ("read error", err)} else {fmt. PRINTLN ("Read succeed", buf)}}func readwithoutalignmentwithdirect (buf []byte) {//Read filefile, err: = OS. OpenFile ("/DEV/SDC", OS. O_rdonly|syscall. O_direct, 0666) if err! = Nil {fmt. Printf ("An error occurred whit file ipening.\n") Return}defer file. Close () buf = buf[2:514]fmt. Println ("Read with buffer", unsafe. Pointer (&buf[0]) _, err = file. ReadAt (buf, MB) If err! = Nil {fmt. PRINTLN ("read error", err)} else {fmt. PRINTLN ("Read succeed", buf)}}func readwithalignmentwithdirect (buf []byte) {//Read filefile, err: = OS. OpenFile ("/DEV/SDD", OS. O_rdonly|syscall. O_direct, 0666) if err! = Nil {fmt. Printf ("An error occurred Whit file ipening.\n ") Return}defer file. Close () buf = buf[512:512+512]fmt. Println ("Read with buffer", unsafe. Pointer (&buf[0]) _, err = file. ReadAt (buf, MB) If err! = Nil {fmt. PRINTLN ("read error", err)} else {fmt. PRINTLN ("Read succeed", BUF)}}

The results of these three functions are as follows.

Read with buffer 0xc42005a002read succeed [4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 

　　

Read with buffer  0xc42005a002read error  read/dev/sdc:invalid argument

　　

Read with buffer 0xc42005a200read succeed [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 

As you can see, since we initially assign the first 20 bits of the slice to 0-20, the other bit assignment defaults to 0, the first write function writes buf[4:516] to/dev/sdb, the second write function fails, and the third write function writes buf[512:512+512] to the /DEV/SDD, so according to the reading results can be seen, our reading function is OK.

Finally, the complete code of the whole test program is given.

Package Mainimport ("FMT" "OS" "Syscall" "unsafe") Func main () {buf: = make ([]byte, 8192) for I: = 0; i <; i++ {Buf[i] = BYTE (i)}fmt. Println ("----------------------------------------") fmt. Println ("White without alignment and DIRECT:") writewithoutalignmentwithoutdirect (BUF) fmt. Println ("----------------------------------------") fmt. Println ("White without alignment but with DIRECT:") writewithoutalignmentwithdirect (BUF) fmt. Println ("----------------------------------------") fmt. Println ("White with alignment and DIRECT:") writewithalignmentwithdirect (BUF) fmt. Println ("----------------------------------------") fmt. PRINTLN ("read without alignment and DIRECT:") readwithoutalignmentwithoutdirect (BUF) fmt. Println ("----------------------------------------") fmt. PRINTLN ("read without alignment but with DIRECT:") readwithoutalignmentwithdirect (BUF) fmt. Println ("----------------------------------------") fmt. Println ("read with alignment and DIRECT:") readwithalignmentwithdirect (BUF)}func writewithoutalignmenTwithoutdirect (buf []byte) {//Open filefile, err: = OS. OpenFile ("/dev/sdb", OS. O_wronly|os. O_create, 0666) if err! = Nil {fmt. Printf ("An error occurred with file opening or creation\n") Return}defer file. Close ()//write Filefmt. Println ("Buffer", unsafe.) Pointer (&AMP;BUF)) fmt. Println ("buffer[0]", unsafe. Pointer (&buf[0])) Buf2: = Buf[4:516]fmt. Println ("Write with buffer", unsafe. Pointer (&buf2[0]) _, err = file. Writeat (buf2, MB) If err! = Nil {fmt. Println ("Write error", err)} else {fmt. Println ("Write Succeed")}}func Writewithoutalignmentwithdirect (buf []byte) {//Open filefile, err: = OS. OpenFile ("/DEV/SDC", OS. O_wronly|os. O_create|syscall. O_direct, 0666) if err! = Nil {fmt. Printf ("An error occurred with file opening or creation\n") Return}defer file. Close ()//write Filefmt. Println ("Buffer", unsafe.) Pointer (&AMP;BUF)) fmt. Println ("buffer[0]", unsafe. Pointer (&buf[0])) Buf2: = Buf[4:516]fmt. Println ("Write with buffer", unsafe. Pointer (&buf2[0]) _, err = file. Writeat (BUF2, +) if Err! = Nil {fmt. Println ("Write error", err)} else {fmt. Println ("Write Succeed")}}func Writewithalignmentwithdirect (buf []byte) {//Open filefile, err: = OS. OpenFile ("/DEV/SDD", OS. O_wronly|os. O_create|syscall. O_direct, 0666) if err! = Nil {fmt. Printf ("An error occurred with file opening or creation\n") Return}defer file. Close ()//write Filefmt. Println ("Buffer", unsafe.) Pointer (&AMP;BUF)) fmt. Println ("buffer[0]", unsafe. Pointer (&buf[0])) Buf2: = Buf[512:512+512]fmt. Println ("Write with buffer", unsafe. Pointer (&buf2[0]) _, err = file. Writeat (buf2, MB) If err! = Nil {fmt. Println ("Write error", err)} else {fmt. Println ("Write Succeed")}}func Readwithoutalignmentwithoutdirect (buf []byte) {//Read filefile, err: = OS. OpenFile ("/dev/sdb", OS. O_rdonly, 0666) if err! = Nil {fmt. Printf ("An error occurred whit file ipening.\n") Return}defer file. Close () buf = buf[2:514]fmt. Println ("Read with buffer", unsafe. Pointer (&buf[0]) _, err = file. ReadAt (buf, MB) If err! = Nil {fmt. Println ("readError ", err)} else {fmt. PRINTLN ("Read succeed", buf)}}func readwithoutalignmentwithdirect (buf []byte) {//Read filefile, err: = OS. OpenFile ("/DEV/SDC", OS. O_rdonly|syscall. O_direct, 0666) if err! = Nil {fmt. Printf ("An error occurred whit file ipening.\n") Return}defer file. Close () buf = buf[2:514]fmt. Println ("Read with buffer", unsafe. Pointer (&buf[0]) _, err = file. ReadAt (buf, MB) If err! = Nil {fmt. PRINTLN ("read error", err)} else {fmt. PRINTLN ("Read succeed", buf)}}func readwithalignmentwithdirect (buf []byte) {//Read filefile, err: = OS. OpenFile ("/DEV/SDD", OS. O_rdonly|syscall. O_direct, 0666) if err! = Nil {fmt. Printf ("An error occurred whit file ipening.\n") Return}defer file. Close () buf = buf[512:512+512]fmt. Println ("Read with buffer", unsafe. Pointer (&buf[0]) _, err = file. ReadAt (buf, MB) If err! = Nil {fmt. PRINTLN ("read error", err)} else {fmt. PRINTLN ("Read succeed", BUF)}}

　　

Use the Go language to bypass page cache read and write files