Golang cumulative hash of performance optimizations

This is a creation in Article, where the information may have evolved or changed.

A lot of times performance problems always happen in some obscure places. In a recent performance analysis, it was discovered that a function that uses temporary strings generated by string concatenation causes memory to rise 40% (120G of memory), and these temporary strings pose a significant burden to the GC, becoming a major performance bottleneck, and these strings act as key to the map. Also must be splicing, so think of directly using the value of hash as the key of the map, and this hash value using the cumulative hash calculation.

The so-called cumulative hash, that is, the hash of the string can be divided into any number of paragraphs, for each successive hash, the result is cumulative, for any kind of segmentation method, the final can get a consistent hash result, such as:, H.hash("hello world") H.hash("hello").hash(" ").hash("world") H.hash("hello wo").hash("rld)" These results should be consistent at the end, With this feature, you can hash multiple strings without stitching

BKDR Hash Implementation

type StringHasherBKDR uint64// NewStringHasherBKDR 创建一个新的 Hasherfunc NewStringHasherBKDR() StringHasherBKDR {    return StringHasherBKDR(0)}// AddStr 增加一个字符串func (bkdr StringHasherBKDR) AddStr(str string) StringHasherBKDR {    val := uint64(bkdr)    for i := 0; i < len(str); i++ {        val = val*131 + uint64(str[i])    }    return StringHasherBKDR(val)}// AddInt 添加一个 int 值func (bkdr StringHasherBKDR) AddInt(i uint64) StringHasherBKDR {    val := uint64(bkdr)    val = val*131 + i    return StringHasherBKDR(val)}// Val 转成 uint64 的值func (bkdr StringHasherBKDR) Val() uint64 {    return uint64(bkdr)}

It's easy to use, too.

hasher := NewStringHasherBKDR()So(hasher.AddStr("hello world").Val(), ShouldEqual, hasher.AddStr("hello").AddStr(" ").AddStr("world").Val())

Reference links

• Code Address: https://github.com/hatlonely/...

Reprint Please specify source
This article link: http://www.hatlonely.com/2018 ...