BOLTDB Study and practice

This is a creation in Article, where the information may have evolved or changed.

Study and practice of Golang Boltdb

1. Installation

Go get Github.com/boltdb/bolt

2. Create and start a database

db, err := bolt.Open("my.db", 0600, nil)

openThe first parameter is the path, if the database does not exist, a database named My.db is created, the second is a file operation, the third parameter is an optional parameter, the internal can be configured for read-only and time-out, etc.
It is especially important to note that because Boltdb is a database of file operation types, it is only possible to write and read in a single point, and if multiple simultaneous operations the latter will be suspended until the former closes, BOLTDB allows only one read-write transaction at a time, but allows multiple read-only transactions at a time. So the data has a strong consistency.

Therefore, individual transactions and all objects created from them, such as buckets, keys, are not thread-safe. With data in multiple concepts one must for each or use lock mechanism to ensure that only one goroutine operation alters the data.
Read-only transactions and read-write things should not normally be opened simultaneously in the same goroutine. This can lead to deadlocks due to the need to periodically remap data files for read and write transactions.

3. Read and Write transactions

BOLTDB Read and Write transaction operations we can use DB.Update() to complete the form such as:

err := db.Update(func(tx *bolt.Tx) error {    ...    return nil})

In the closure fun, returns nil at the end to commit the transaction. You can also roll back a transaction at any point by returning an error. All database operations are allowed in a read-write transaction.
Always pay attention to err back, as it will report all disk failures that cause your transaction not to complete.

4. Read and write things in bulk

Every new thing needs to wait for the end of the last thing, and this kind of overhead can be DB.Batch() done through batch processing.

err := db.Batch(func(tx *bolt.Tx) error {    ...    return nil})

In batch processing, if a transaction fails, the batch calls the function function multiple times to return success. If the midway fails, the entire transaction is rolled back.

5. Read-Only transactions

Read-only transactions can be used DB.View() to complete

err := db.View(func(tx *bolt.Tx) error {    ...    return nil})

Operations that do not change the data can be done through a read-only transaction, and you can only retrieve buckets, retrieve values, or replicate the database in a read-only transaction.

6. Start a transaction

DB.Begin()The start function is contained in db.update and Db.batch, which starts the transaction and returns the result of the transaction, which is the boltdb recommended way, sometimes you may need to start things manually you can use Tx.Begin() to start, remember not to forget to close the transaction.

// Start a writable transaction.tx, err := db.Begin(true)if err != nil {    return err}defer tx.Rollback()// Use the transaction..._, err := tx.CreateBucket([]byte("MyBucket"))if err != nil {    return err}// Commit the transaction and check for error.if err := tx.Commit(); err != nil {    return err}

7. Using Buckets

A bucket is a collection of key/value pairs in a database. All keys in a bucket must be unique. You can DB.CreateBucket() create a bucket using:

db.Update(func(tx *bolt.Tx) error {    b, err := tx.CreateBucket([]byte("MyBucket"))    if err != nil {        return fmt.Errorf("create bucket: %s", err)    }    return nil})

You can also be useful Tx.CreateBucketIfNotExists() to create buckets, the function will first determine whether the bucket already exists that is created, delete buckets can be used Tx.DeleteBucket() to complete

8. Use K-v to

Store key value pairs into the bucket can be used Bucket.Put() to complete:

db.Update(func(tx *bolt.Tx) error {    b := tx.Bucket([]byte("MyFriendsBucket"))    err := b.Put([]byte("one"), []byte("zhangsan"))    return err})

To get the key value Bucket.Get() :

db.View(func(tx *bolt.Tx) error {    b := tx.Bucket([]byte("MyFriendsBucket"))    v := b.Get([]byte("one"))    fmt.Printf("The answer is: %s\n", v)    return nil})

get()The function does not return an error because its run is guaranteed to work (unless there is some kind of system failure). If the key exists, it returns its value. If it does not exist, then it will return nil.
It is also important to note that when a transaction is opened, the value returned by the get is only valid, and if you need to use that value for other transactions, you can copy copy it to the other byte slice

9. Barrel Self-increment

With nextsequence() functionality, you can have boltdb generate sequences as a unique identifier for your key-value pairs. See the example below.

func (s *Store) CreateUser(u *User) error {    return s.db.Update(func(tx *bolt.Tx) error {        // 创建users桶        b := tx.Bucket([]byte("users"))        // 生成自增序列        id, _ = b.NextSequence()        u.ID = int(id)        // Marshal user data into bytes.        buf, err := json.Marshal(u)        if err != nil {            return err        }        // Persist bytes to users bucket.        return b.Put(itob(u.ID), buf)    })}// itob returns an 8-byte big endian representation of v.func itob(v int) []byte {    b := make([]byte, 8)    binary.BigEndian.PutUint64(b, uint64(v))    return b}type User struct {    ID int    ...}

10. Iteration Key

Boltdb stores the key in the byte sort order in the bucket. This makes the order iterations on these keys very fast. To traverse the key, we will use the cursor Cursor() :

db.View(func(tx *bolt.Tx) error {    // Assume bucket exists and has keys    b := tx.Bucket([]byte("MyBucket"))    c := b.Cursor()    for k, v := c.First(); k != nil; k, v = c.Next() {        fmt.Printf("key=%s, value=%s\n", k, v)    }    return nil})

Cursors Cursor() allow you to move to a specific point in the list of keys and move forward or backward through the action key one at a time.
The following functions are available on the cursor:

First()  移动到第一个健.Last()   移动到最后一个健.Seek()   移动到特定的一个健.Next()   移动到下一个健.Prev()   移动到上一个健.

Each of these functions returns a signature that contains (key []byte, Value []byte). When you have the end of the cursor iteration, next () will return a nil. Before calling next () or prev (), you must seek a location using first (), last (), or seek (). If you do not seek a location, these functions will return a nil key.
During the iteration, if the key is nonzero, but the value is 0, it means that the key points to a bucket instead of a value. Use bucket. Bucket () to access the child bucket.

11. Prefix scan

To iterate through a key prefix, you can combine seek() and bytes.hasprefix() :

db.View(func(tx *bolt.Tx) error {    // Assume bucket exists and has keys    c := tx.Bucket([]byte("MyBucket")).Cursor()    prefix := []byte("1234")    for k, v := c.Seek(prefix); bytes.HasPrefix(k, prefix); k, v = c.Next() {        fmt.Printf("key=%s, value=%s\n", k, v)    }    return nil})

12. Range Scanning

Another common use case is the scope of the scan, such as the time range. If you use a suitable time code, such as rfc3339, you can then query data for a specific date range:

db.View(func(tx *bolt.Tx) error {    // Assume our events bucket exists and has RFC3339 encoded time keys.    c := tx.Bucket([]byte("Events")).Cursor()    // Our time range spans the 90's decade.    min := []byte("1990-01-01T00:00:00Z")    max := []byte("2000-01-01T00:00:00Z")    // Iterate over the 90's.    for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {        fmt.Printf("%s: %s\n", k, v)    }    return nil})

13. Loop through each

If you know that you have a key in your bucket, you can use it ForEach() to iterate:

db.View(func(tx *bolt.Tx) error {    // Assume bucket exists and has keys    b := tx.Bucket([]byte("MyBucket"))    b.ForEach(func(k, v []byte) error {        fmt.Printf("key=%s, value=%s\n", k, v)        return nil    })    return nil})

14. Nested Buckets

You can also store a bucket in a key to create a nested bucket:

func (*Bucket) CreateBucket(key []byte) (*Bucket, error)func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)func (*Bucket) DeleteBucket(key []byte) error

15. Database backup

Boltdb is a single file, so it's easy to back up. You can use TX.writeto() a function to write a consistent database. If this function is called from a read-only transaction, it performs a hot backup without blocking the read and write operations of the other databases.
By default, it uses a regular file handle that takes advantage of the operating system's page cache. For information about optimizing larger than RAM datasets, see the Tx documentation.
A common use case is to make a backup on HTTP so that you can use a cURL tool like this to make a database backup:

func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {    err := db.View(func(tx *bolt.Tx) error {        w.Header().Set("Content-Type", "application/octet-stream")        w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)        w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))        _, err := tx.WriteTo(w)        return err    })    if err != nil {        http.Error(w, err.Error(), http.StatusInternalServerError)    }}

You can then use this command to back up:
$ curl http://localhost/backup > my.db
Or you can open your browser to http://localhost/backup and it will download automatically.
If you want to back up to another file, you can use TX.copyfile() accessibility features.

16. Statistics

The database maintains a running count of many internal operations that run, so you can better understand what's going on. By capturing a snapshot of the data, we can see what has been done within that timeframe.
For example, we can start logging statistics every 10 seconds in a goroutine:

go func() {    // Grab the initial stats.    prev := db.Stats()    for {        // Wait for 10s.        time.Sleep(10 * time.Second)        // Grab the current stats and diff them.        stats := db.Stats()        diff := stats.Sub(&prev)        // Encode stats to JSON and print to STDERR.        json.NewEncoder(os.Stderr).Encode(diff)        // Save stats for the next loop.        prev = stats    }

17. Read-only mode

Sometimes it is useful to create a shared read-only BOLTDB database. For this, set the OPTIONS.READONLY flag when the database is opened. Read-only mode uses a shared lock to allow multiple processes to read from the database, but it blocks any process that opens the database in a read-write manner.

db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})if err != nil {    log.Fatal(err)}

18. Mobile-side Support (ios/android)

Boltdb is able to run tools combined with feature gomobile on mobile devices. Create a struct that contains your database logic and reference a bolt.db with initialization contstructor required in the file path, the database file will be stored. With this approach, both Android and iOS do not require additional permissions or cleanup.

func NewBoltDB(filepath string) *BoltDB {    db, err := bolt.Open(filepath+"/demo.db", 0600, nil)    if err != nil {        log.Fatal(err)    }    return &BoltDB{db}}type BoltDB struct {    db *bolt.DB    ...}func (b *BoltDB) Path() string {    return b.db.Path()}func (b *BoltDB) Close() {    b.db.Close()}

The database logic should be defined as a method in this wrapper structure.
To initialize this structure from a native language (two platforms now synchronize local storage with the cloud). These fragments disable the functionality of the database file):
Android

String path;if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){    path = getNoBackupFilesDir().getAbsolutePath();} else{    path = getFilesDir().getAbsolutePath();}Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)

Ios

- (void)demo {    NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,                                                          NSUserDomainMask,                                                          YES) objectAtIndex:0];    GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);    [self addSkipBackupAttributeToItemAtPath:demo.path];    //Some DB Logic would go here    [demo close];}- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString{    NSURL* URL= [NSURL fileURLWithPath: filePathString];    assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);    NSError *error = nil;    BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]                                  forKey: NSURLIsExcludedFromBackupKey error: &error];    if(!success){        NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);    }    return success;}

19. Viewing tools

1. Download tool
go get github.com/boltdb/boltd
Then compile the cmd under the main file to generate the executable file renamed to Boltd
Copy the Boltd to the *.db sibling directory and do the following:

Then open the Web site:

2. Command-line tools
Https://github.com/hasit/bolter
BOLTDB Basic Learning for the time being so much, the next chapter begins to practice