The road of Golang optimization--bitset

This is a creation in Article, where the information may have evolved or changed.

Write in front

The collection of such data structures is often handled during development, and simple points are handled using the built-in map implementation. But if you're dealing with large amounts of data, the problem with using map to take up memory is obvious. Memory is high and there are some problems with columns, and this is not the start. Also, a large amount of data is stored in a map, and the hash algorithm used to find it can be very expensive. It is time to consider optimizing the data structure. Before browsing awesome-go, we found a data structure called bitset, and introduce it today.

Bitset Introduction

First, this is a data structure. From the name set it is not difficult to find that this is a collection of data structures. Bit is also more understood, through set is implemented by bit. If you need a collection, just the elements in the collection are positive integers, then this is the right thing to do.

Example

import "github.com/willf/bitset"var b bitset.BitSet // 定义一个BitSet对象b.Set(10).Set(11) // 给这个set新增两个值10和11if b.Test(1000) { // 查看set中是否有1000这个值（我觉得Test这个名字起得是真差劲，为啥不叫Exist）b.Clear(1000) // 情况set}for i,e := v.NextSet(0); e; i,e = v.NextSet(i + 1) { // 遍历整个Set   fmt.Println("The following bit is set:",i);}if B.Intersection(bitset.New(100).Set(10)).Count() > 1 { // set求交集fmt.Println("Intersection works.")}

This package has been fully functional, and the complete documentation can be consulted on its godoc. I use these packages, in addition to the basic functions (for the collection, is to delete and change these), there is a convenient debugging. Bitset internal preservation of the numbers are stored, if the debugging is the bitset of the internal data to me to see, I also do not understand, fortunately this package provides a String() method, you can put the data I set to return the form of the string, Bang Bang.

Implementation principle

It is my style to study the principle of implementation. Probably say the principle. A set of positive integers can be placed inside a large integer, using bits to represent numbers. For example 1001 , you can represent 0 and 2 of these two numbers. Using a bit instead of one int can greatly reduce the memory footprint. But an integer is the maximum of 64 bits, that is, the largest representation of the number is 64, so you can use multiple int splicing form to represent large integers.

Bitset's internal data structure, very kind with wood:

type BitSet struct {length uint // set的大小set    []uint64 // 这个就会被用来表示一个大整数}

Use the following test code to explore the internal implementation:

var b bitset.BitSet // 定义一个BitSet对象fmt.Println(b.Bytes())b.Set(0)fmt.Println(b.Bytes(),0)b.Set(10) // 给这个set新增两个值10fmt.Println(b.Bytes(),0,10)b.Set(64)fmt.Println(b.Bytes(),0,10,64)if b.Test(1000) { // 查看set中是否有1000这个值（我觉得Test这个名字起得是真差劲，为啥不叫Exist）b.Clear(1000) // 情况set}

Output: [][1] 0[1025] 0 10[1025 1] 0 10 64

• The new Bitset,set is empty[]
• Put in a 0, with the first digit, that is,0x00000001
• Put in 10, internal structure0x00000041
• Put in 64, this time an integer has not been saved, the internal structure is 0x00000041 and 0x00000001 . The set in this array, from the back of the data represented by the increase in sequence, but in the uint64 interior, is from the low start, low to indicate a small number.

Comparison with other data structures

Represents a set of positive integers, Golang has a lot of ways, it map can be, of course, this is the worst choice, the first is the waste of memory, followed by each search also involves the hash calculation, although the theoretical HashMap complexity is O (1), In fact, compared with Bitset is completely slag. In addition, Bitset have to upgrade version of Roaring is also a good choice. If you want to save the data is 10000000000 this level, then with Bitset there will be a low waste of memory, roaring can be used to compress space.

  import ("Testing" "github.com/roaringbitmap/roaring" "Github.com/willf/bitset") func Benchmarkmap (b *testing. b) {var b = make (Map[int]int8, 3) b[10] = 1b[11] = 1for I: = 0; i < B.N; i++ {If _, exists: = b[1]; exists {}if _, exist S: = b[11]; exists {}if _, exists: = b[1000000]; exists {}}}func Benchmarkbitset (b *testing). b) {var b bitset. Bitsetb.set (10). Set (one) for I: = 0; i < B.N; i++ {if b.test (1) {}if b.test (one) {}if b.test (1000000) {}}}func benchmarkroaring (B *testing). b) {for i: = 0; i < B.N; i++ {b: = roaring. Bitmapof (Ten, one) if B.containsint (1) {}if b.containsint (one) {}if b.containsint (1000000) {}}}$ go test-bench=.*-benchmem B EnchmarkMap-2 50000000 28.4 ns/op 0 b/op 0 allocs/opbenchmarkbitset-2 2 000000000 1.86 ns/op 0 b/op 0 allocs/opbenchmarkroaring-2 3000000 49 2 ns/op b/op 6 allocs/op  

Conclusion

If you are comparing consecutive nonnegative integers, it is recommended to solve the collection problem with Bitset. Of course specific problems specific analysis.

Please refer to the complete source code in this article.

Original link: Golang Optimization road--bitset, reprint please specify the source!