Using go for block chain Development 3: Persistence and Cli_go

Original address

The original address https://jeiwan.cc/posts/building-blockchain-in-go-part-3/introduction

So far, we've done a block chain with a work-proof system, so it can be dug. Our implementation is getting closer to the fully functional block chain, but it lacks some important features. Today we're going to save the block chain in a database and then do a simple command-line tool to manipulate it. In essence, a block chain is a distributed database. We temporarily ignore the "distributed" section and focus on the "Database" section. Database selection

We don't have a database in our current implementation, but we create a chunk chain and save it in memory every time we execute the program. We cannot reuse a block chain, nor can we share it with others, so we need to have it on disk.

Which database do we need? In fact, any one is OK. The use of specific databases is not mentioned in the Bitcoin white paper, so what database is used depends on the developer. The universal Bitcoin implementation reference is the Bitcoin Core originally released by Satoshi Nakamoto, which used Leveldb (although it was released 2012 years ago). What we are going to use is ...

Boltdb

Because:
It is minimalist and easy to use.
Use GO implementation. You do not need to run the server. Allows you to create the data structure that we want.

The following is an excerpt from Boltdb's readme on GitHub

Bolt is a pure go Key/value store inspired by Howard Chu ' s Lmdb project. The goal of the "project is" to provide a simple, fast, and reliable database for projects this don ' t require a full databas E server such as Postgres or MySQL.

Since Bolt is meant to being used as such a low-level piece of functionality, simplicity is key. The API would be small and getting values and setting values. That ' s it.

Sounds very much in line with our needs. Take a minute to get a closer look.

Boltdb is a database of key-value pairs stored, which means that it does not have tables in SQL RDBMS (such as mysql,postgresql, etc.), no rows, no columns. Instead, the data is stored in the form of key-value pairs (like a map in Golang). Key-value pairs are stored in buckets (bucket) and are intended to group similar key-value pairs (similar to tables in an RDBMS). Therefore, getting a value requires knowing its key and the bucket in which it resides.

An important point about boltdb is that there are no data types: Both the key and the value are byte arrays. Because we want to store the go structure (especially block), we need to serialize it, such as implementing a mechanism that converts the go structure to a byte array and can restore it from byte data. We'll use Encoding/gob for this, but JSON, XML, Protocolbuffers, and so on are all OK. We use Encoding/gob because it is simple and is in the Go standard library.

Database structure

Before we start implementing the persistence logic, let's decide how to store the data in the database. We'll refer to the Bitcoin core approach.

Simply put, Bitcoin Core uses two buckets to save data:
The block holds metadata for all the blocks in the description chain. Chainstate the state of the chain, that is, the current output of all outstanding transactions and some metadata.

In addition, blocks are stored in separate files on disk. This is done for performance purposes: reading a single chunk does not require loading all (or more) into memory. We will not achieve this.

The following key value pairs are in the BLOCK:
' B ' + 32-byte chunk hash-> chunk index record ' F ' + 4-byte file number-> file information record ' L ' + 4 byte file number-> the last block with the file number ' R ' + 1-byte Boolean value-> whether the index ' F ' + is being rebuilt 1-byte ID + ID string-> 1-byte boolean value: A transaction hash-> transaction index record that can be placed on and off for various identifiers ' t ' + 32 bytes

The key values in the Chainstate are as follows:
' C ' + 32-byte transaction hash-> The transaction's unhandled output record ' B '-> 32-byte chunk hash: This hash depends on the database that describes the unhandled transaction output

(For more information here)

Since we have not yet traded, we now use only one blocks bucket. In addition, as we have just said, we will save the entire database to a single file instead of saving each chunk in a separate file. Therefore, the content associated with the file number is not needed. Now, the key->value we use are these:
32-byte chunk hash-> the hash value of the last block in a block structure (serialized) ' L '-> chain

That's all we need to implement the persistence mechanism. Serialization of

As previously mentioned, only byte arrays can be used in boltdb, and we are going to store the block structure in the database. We use Encoding/gob to serialize the structure.

We're going to implement Block's Serialize method (for simplicity or for skipping error handling):

Func (b *block) Serialize () []byte {
    var result bytes. Buffer
    Encoder: = gob. Newencoder (&result)

    err: = Encoder. Encode (b) return result

    . Bytes ()
}

This code is intuitive: first declare a buffer to hold the serialized data, then initialize a GOB encoder and encode the block, and the result is returned as a byte array.

Next we need a deserialization method that receives an array of bytes and returns a block. This is not a method but a separate function:

Func Deserializeblock (d []byte) *block {
    var block block

    Decoder: = Gob. Newdecoder (bytes. Newreader (d))
    err: = decoder. Decode (&block) return

    &block
}

This is the serialization section. Persistence of

Let's start with the Newblockchain method. At present it is to create a blockchain instance and add the Genesis block. And what we want is this:

Open a database file. Check if there is a block chain. If there are:
Create a new blockchain instance. Set the end of the blockchain instance to the last block stored in the database if there is no block chain:
Create the Genesis block into the database and put the Hachicun of the Genesis block as the last block hash store to create a blockchain instance of the tail point to the Genesis block

The code is as follows:

Func Newblockchain () *blockchain {
    var tip []byte
    db, err: = Bolt. Open (DBFile, 0600, nil)

    err = db. Update (func (TX *bolt). TX) Error {
        b: = Tx. Bucket ([]byte (Blocksbucket))

        if b = = Nil {
            Genesis: = Newgenesisblock ()
            B, err: = Tx. Createbucket ([]byte (Blocksbucket))
            err = B.put (Genesis. Hash, Genesis. Serialize ())
            err = B.put ([]byte ("L"), Genesis. Hash)
            tip = Genesis. Hash
        } else {
            tip = B.get ([]byte ("L")
        } return

        nil
    })

    BC: = blockchain{tip, db}

    Return &BC
}

Let's take a look at this code again.

DB, err: = Bolt. Open (DBFile, 0600, nil)

This is the standard way to open boltdb files. It is worth noting that the file does not return an error if it does not exist.

Err = db. Update (func (TX *bolt). Tx) Error {
...
})

In Boltdb, operations on the database are performed in a transaction. Transactions are divided into two types: read-only and read-write. Because we wanted the Genesis block to be stored in the database, where we opened a read-write transaction (db. Update (...) ）。

B: = Tx. Bucket ([]byte (Blocksbucket))

if b = = Nil {
    Genesis: = Newgenesisblock ()
    B, err: = Tx. Createbucket ([]byte (Blocksbucket))
    err = B.put (Genesis. Hash, Genesis. Serialize ())
    err = B.put ([]byte ("L"), Genesis. Hash)
    tip = Genesis. Hash
} else {
    tip = B.get ([]byte ("L")
}

The code above is the core of this function. We get the bucket that holds the BLOCK: Read the key L if it exists, create the Genesis block if it does not exist, build the bucket, save the Genesis block, and then update the L key that holds the last chunk hash in the chain.

Also, note the new way we create blockchain:

BC: = blockchain{tip, db}

Instead of storing all the blocks in, we save only the end of the saved block chain. A database link is also saved because we want it to be open as soon as the program is running. So now the blockchain structure is like this:

Type blockchain struct {
    tip []byte
    db  *bolt. DB
}

The next update is the Addblock method: Adding blocks now is no longer as simple as adding elements to an array. To save chunks to the database from now on:

Func (BC *blockchain) addblock (data string) {
    var lasthash []byte

    err: = Bc.db.View (func (TX *bolt.) TX) Error {
        b: = Tx. Bucket ([]byte (Blocksbucket))
        Lasthash = B.get ([]byte ("L")) return

        nil
    })

    Newblock: = Newblock (data, Lasthash)

    err = Bc.db.Update (func (TX *bolt. TX) Error {
        b: = Tx. Bucket ([]byte (Blocksbucket))
        err: = B.put (Newblock.hash, Newblock.serialize ())
        err = B.put ([]byte ("L"), Newblock.hash)
        Bc.tip = Newblock.hash return

        nil
    })
}

1.1 Points to see:

ERR: = Bc.db.View (func (TX *bolt). TX) Error {
    b: = Tx. Bucket ([]byte (Blocksbucket))
    Lasthash = B.get ([]byte ("L")) return

    Nil
})

This is another type of (read-only) BOLTDB transaction. We have the hash value of the last block in the stored database to excavate a new chunk hash.

Newblock: = Newblock (data, Lasthash)
b: = Tx. Bucket ([]byte (Blocksbucket))
err: = B.put (Newblock.hash, Newblock.serialize ())
err = B.put ([]byte ("L"), Newblock.hash)
Bc.tip = Newblock.hash

After digging into the new block, we store the serialized data in the database and update the hash value of the L key.

Complete. It's not hard, is it? Check block chain

Now all the blocks are in the database, so we can reopen a block chain and add a new zone block to it. But after that we lost a good feature: We can't print chunk information because we no longer save chunks to the array. We're going to fix this flaw!

Boltdb allows you to iterate through all the keys in a bucket, but the keys are stored in byte order, and we want to print them in the order of the blocks in the block chain. In addition, because we do not want to load all blocks into memory (our block chain database can be very large ...). We pretend it's big) and we'll read the blocks one by one. To do this, you need a block chain iterator:

Type blockchainiterator struct {
    currenthash []byte
    db          *bolt. DB
}

We want to iterate over all the blocks in the block chain and create an iterator that holds the chunk hash of the current iteration and a database link. Given the latter, an iterator is logically attached to a block chain (saving a blockchain instance of a database link), so it is created in a blockchain method:

Func (BC *blockchain) iterator () *blockchainiterator {
    BCI: = &blockchainiterator{bc.tip, bc.db} return

    BCI
}

Note that the iterator initially points to the end of the block chain, so the blocks are fetched from the top down, from the new to the old. In fact, choosing an end means "voting" for a block chain. A block chain can have multiple branches, the longest of which is considered to be the primary branch. After getting an end (which can be any chunk of the block chain) we can reconstruct the whole block chain and get its length and the work it needs to build. This also means that an end is an identifier for a block chain.

Blockchainiterator do one thing: return to the next block on the block chain.

Func (i *blockchainiterator) Next () *block {
    var block *block
    Err: = I.db.view (func (TX *bolt.) TX) Error {
        b: = Tx. Bucket ([]byte (Blocksbucket))
        Encodedblock: = B.get (i.currenthash) Block
        = Deserializeblock (Encodedblock) return
        Nil
    })
    I.currenthash = block. Prevblockhash return block
}

The above is the database section. Cli

So far our implementations have provided no interface for interacting with the program: we simply executed the Newblockchain, BC, in the main function. Addblock. It's time to improve on this. We want the following command:

Blockchain_go Addblock "Pay 0.031337 for a coffee" blockchain_go

All command-line-related operations are handled by the CLI structure:

Type CLI struct {
    BC *blockchain
}

Its "portal" is the Run function:

Func (CLI *cli) Run () {
    Cli.validateargs ()

    addblockcmd: = flag. Newflagset ("Addblock", flag. Exitonerror)
    printchaincmd: = flag. Newflagset ("Printchain", flag. Exitonerror)

    Addblockdata: = addblockcmd.string ("Data", "", "Block data")

    switch OS. ARGS[1] {case
    "Addblock":
        Err: = Addblockcmd.parse (OS. Args[2:] Case
    "Printchain":
        Err: = Printchaincmd.parse (OS. Args[2:])
    default:
        cli.printusage ()
        OS. Exit (1)
    }

    if addblockcmd.parsed () {
        if *addblockdata = = "" {
            addblockcmd.usage ()
            OS. Exit (1)
        }
        cli.addblock (*addblockdata)
    }

    if printchaincmd.parsed () {
        cli.printchain ()
    }
}

We use the flag package in the standard library to parse the command-line arguments:

Addblockcmd: = flag. Newflagset ("Addblock", flag. Exitonerror)
printchaincmd: = flag. Newflagset ("Printchain", flag. Exitonerror)
Addblockdata: = addblockcmd.string ("Data", "", "Block data")

First, the key two subcommand, Addblock and Printchain, and then add the-data identity to the former. Printchain is not identified.

Switch OS. ARGS[1] {case
"Addblock":
    Err: = Addblockcmd.parse (OS. Args[2:] Case
"Printchain":
    Err: = Printchaincmd.parse (OS. Args[2:])
default:
    cli.printusage ()
    OS. Exit (1)
}

Next, check the user-supplied command and resolve the associated flag subcommand.

If addblockcmd.parsed () {
    if *addblockdata = = "" {
        addblockcmd.usage ()
        OS. Exit (1)
    }
    cli.addblock (*addblockdata)
}

if printchaincmd.parsed () {
    cli.printchain ()
}

Then check which subcommand is parsed and perform the related functions.

Func (CLI *cli) addblock (data string) {
    cli.bc.AddBlock (data)
    FMT. Println ("success!")
}

Func (CLI *cli) Printchain () {
    BCI: = Cli.bc.Iterator () for

    {block
        : = BCI. Next ()

        FMT. Printf ("Prev. Hash:%x\n", block. Prevblockhash)
        FMT. Printf ("Data:%s\n", block.) Data)
        FMT. Printf ("Hash:%x\n", block. Hash)
        Pow: = newproofofwork (block)
        FMT. Printf ("PoW:%s\n", StrConv.) Formatbool (POW. Validate ())
        FMT. PRINTLN ()

        If Len (block. Prevblockhash) = = 0 {break}}}

This code is similar to what we wrote before. The only difference is that now we use a blockchainiterator to iterate over chunks in the block chain.

Also, do not forget to modify the main function:

Func Main () {
    BC: = Newblockchain ()
    defer bc.db.Close ()

    CLI: = CLI{BC}
    CLI. Run ()
}

It is worth mentioning that no matter what the command-line arguments are, a new blockchain will be created.

So much. Check to see if everything is OK:

$ blockchain_go printchain No existing blockchain found.
Creating a new one ... Mining The block containing "Genesis block" 000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b Prev. Hash:Data:Genesis block hash:000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b pow:true $ blockchain _go addblock-data ' send 1 BTC to Ivan ' Mining The block containing ' send 1 BTC to Ivan ' 000000d7b0c76e1001cdc1fc866b95a48

1d23f3027d86901eaeb77ae6d002b13 success! $ blockchain_go addblock-data "Pay 0.31337 BTC for a coffee" Mining the blocks containing "Pay 0.31337 BTC for a coffee" 0

00000aa0748da7367dec6b9de5027f4fae0963df89ff39d8f20fd7299307148 success! $ blockchain_go printchain Prev. Hash:000000d7b0c76e1001cdc1fc866b95a481d23f3027d86901eaeb77ae6d002b13 Data:pay 0.31337 BTC for a coffee hash:000000aa0748da7367dec6b9de5027f4fae0963df89ff39d8f20fd7299307148 pow:true Prev. hash:000
000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b data:send 1 BTC to IvanHash:000000d7b0c76e1001cdc1fc866b95a481d23f3027d86901eaeb77ae6d002b13 pow:true Prev. Hash:Data:Genesis Block hash:0 00000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b Pow:true

(Here is the sound of beer) conclusion

Next time we will implement the address, wallet, transaction (possible). So keep your eye on it.

Links: Full source Bitcoin core data storage BOLTDB ENCODING/GOB flag