Blockchain + Big data: EOS storage

When it comes to blockchain storage, it's easy to think of its chained storage structure, but the blockchain has evolved from Bitcoin to today's most popular EOS, and technology has been evolving for 10 of years. The current EOS storage, in addition to confirming the chained storage of the structure, has made a great progress in state storage, especially after the introduction of MongoDB plugin, which can be used to hook up the limited State Library to big data shuttles. This article will introduce the storage technology of EOS comprehensively.

EOS storage, Merkle tree,mongodb,chainbase, source learning, Context_free_actions

The chain storage structure of EOS

The EOS block data structure is as follows:

Field	explanation
Timestamp	Time stamp
Producer	Producers
Confirmed	Number of producers confirmed
Previous	The ID of the previous chunk of the chain structure
Transaction_mroot	Trading Merkelshigen
Action_mroot	Action Merkelshigen
Schedule_version	Producer version Sort number
New_producers	Next producer
Header_extensions	Area Header Extension Field
Producer_signature	Block signature, signed by producer
Transactions	Block Package transaction content, is the array structure, can be multiple
Block_extensions	Chunk extension fields
Id	Current Block ID
Block_num	Current block height
Ref_block_prefix	Chunk header for referencing chunks

Merkle Tree

The evolution of the Merkel tree is the hash and hash tree and the Merkle tree, all of which exist to address data consistency, with the following meanings:

• Hash is a technology that we all know, it can generate a hash value for a file or other data, when we download a file, usually on the download page to see the hash value of the file and the hash value of the algorithm, after downloading, We can be local to the entire file for the same hash algorithm to get the hash value, and then with the hash value on the page to compare, if the same, the file is complete, is the source file on the Web page, if not match the file is corrupted, modified or incomplete.
• is still the file integrity of the verification requirements, when this file is particularly large, the hash algorithm for this file is a huge energy consumption, so you can cut the file into a lot of small pieces, each small piece has a hash, And then the hash value of all the small pieces together again hash algorithm to get is the root hash. In this way, when we download large files, we will first download a hash list containing the root hash, by verifying the root hash can determine the correctness of the hash list, to determine the correct hash list, and then download a small piece of file and verify the hash one by one, When a small block hash is found to be mismatched, it can be downloaded separately without having to download it all again. The structure of the hash list is actually a root hash, and the small block hashs is a tree with a height of 2 for the leaf nodes.
• Merkle tree is actually an optimization of the hash list, which greatly improves performance. Its structure is a binary tree (also can be a multi-fork tree, the key point of performance optimization is that its height is greater than or equal to 2), each node up to only two sub-nodes, only the leaf node is based on small pieces of file hash, the parent of every two adjacent leaf node is the parent hash of the two hash, If the total number of leaf nodes is singular, then there will be one remaining, step up, and eventually there will be a root node, which is Merkle root. In this way, when we download large files, we will download a merkle tree first, check from the leftmost node, step up, and complete the Merkle tree check. This is different from the above hash tree is, as long as the left-most adjacent two leaf node hash value and their parent node hash through the match, you can immediately start to download the corresponding file blocks of the two leaf nodes, parallel, and then verify the other leaf nodes, which improves performance, You do not have to validate the full Merkle tree before downloading the file.

Merkle Tree and Blockchain

The block data structure above contains two fields related to Merkle tree:

• Transaction_mroot, the Transactions field in a chunk can contain multiple transactions, and the Transaction_mroot in the block is the Merkle Root of all the packaged transactions within that block, which can be used to verify the correctness of each trade. If the chunk does not contain any transactions, the value of this field is 0000000000000000000000000000000000000000000000000000000000000000. When the node synchronizes the data, the Merkle tree of the transaction is downloaded and verified by Merkle root, rather than downloading all the trading principals, thus saving the amount of data on the light node.
• Action_mroot, create an action root based on all distributions, and evaluate when a transaction is received within a chunk. Used in the light client calibration, the function ibid.

Action_mroot is always valued, even if Transaction_mroot is 0, because the block itself is also an action action Onblock, which invokes the Onblock function of the system contract. TODO: Source Code Analysis

/***  At the start of each block we notify the system contract with a transaction that passes in*  the block header of the prior block (which is currently our head block)*/signed_transaction get_on_block_transaction(){  action on_block_act;  on_block_act.account = config::system_account_name;  on_block_act.name = N(onblock);  on_block_act.authorization = vector<permission_level>{{config::system_account_name, config::active_name}};  on_block_act.data = fc::raw::pack(self.head_block_header());  signed_transaction trx;  trx.actions.emplace_back(std::move(on_block_act));  trx.set_reference_block(self.head_block_id());  trx.expiration = self.pending_block_time() + fc::microseconds(999'999); // Round up to nearest second to avoid appearing expired  return trx;}

Context_free_actions

Through the nouce action of the Eosio.null account, unsigned data can be packaged into the Context_free_action field, and the result block information is as follows:

evsward@evsward-tm1701:~/work/src/github.com/eos/tutorials/bios-boot-tutorial$ Cleos--wallet-url/http  127.0.0.1:6666--url http://127.0.0.1:8000 Get block 440{"timestamp": "2018-08-14t08:47:09.000", "producer": "Eosio", "Confirmed": 0, "previous": "000001b760e4a6610d122c5aa5d855aa49e29f3052ac3e40b9e1ef78e0f1fd02", "Transaction_mroot ":" 32cb43abd7863f162f4d8f3ab9026623ea99d3f8261d2c8b4d8bf920ab97e3d1 "," Action_mroot ":" 09afeaf40d6988a14e9e92817d2ccf4023b280075c99f13782a6535ccc58cbb0 "," Schedule_version ": 0," new_producers ": null," Header_extensions ": []," producer_signature ":" Sig_k1_ K2efdzbxcg3hmqzpzpulymiesrcipmthdensqdyfgchumfemc3pntxtqiup5vunmyb7qmh18fbdmunksc7jgcm1tspfbaj "," Transactions " : [{"Status": "Executed", "Cpu_usage_us": 290, "net_usage_words": +, "Trx": {"id": "d7484374 9d1e255f13572b7a3b95af9ddd6df23d1d0ad19d88e1496091d4be2b "," signatures ": [" SIG_K1_KVZWG3QRH6ZMEMPNSVAXPPQ A42hf4tdpv5cqwqo7ey4osu7nmrefwg7gdsdcnuhHhmh1ewtvamv1z9bqttvvqnsxisgawg "]," compression ":" None "," Packed_context_free_data ":" "," Co Ntext_free_data ": []," Packed_trx ":" 8497725bb601973ea96f0000000100408c7a02ea3055000000000085269d000706686168616861010082c95865ea3055000000000000806b010082c95          865ea305500000000a8ed3232080000000000d08cf200 "," transaction ": {" Expiration ":" 2018-08-14t08:49:08 ", "Ref_block_num": 438, "Ref_block_prefix": 1873362583, "Max_net_usage_words": 0, "Max_cpu_u              Sage_ms ": 0," delay_sec ": 0," context_free_actions ": [{" Account ":" Eosio.null ",          "Name": "Nonce", "Authorization": [], "Data": "06686168616861"}],                  "Actions": [{"Account": "Eosiotesta1", "name": "HI", "authorization": [{           "Actor": "Eosiotesta1", "permission": "Active"}   ], "data": {"user": "Yeah"}, "Hex_data": "0000000000d08cf2" }], "Transaction_extensions": []}}], "Block_extensions": [], "id": "0000 01b8d299602b289a9194bd698476c5d39c5ad88235460908e9d43d04edc8 "," Block_num ":" Ref_block_prefix ": 2492570152}

The content of normal actions is the call of Hi Smart Contract, and the context_free_action contains unsigned data, which is the pattern after the Digital digest. Operation in the source code:

// lets also push a context free action, the multi chain test will then also include a context free action("context_free_actions", fc::variants({    fc::mutable_variant_object()       ("account", name(config::null_account_name))       ("name", "nonce")       ("data", fc::raw::pack(v))    }) );

Statedb of EOS

Let's imagine a scenario:

A account transfer to B account 100 sys, how to view the balance of a account?

For those of us who do not know when these actions occur, how do we do it:

• The first is to scan the block from the beginning of the transaction, the action within the trade, until you find the account A is created by the block number corresponding to the action.
• Continue scanning from this block number to record and count all the actions for all a account transfers, including revenue and expenses.
• Calculates the current balance of a.

The above steps are easy to error and cumbersome, every time the balance of the query to repeat these operations is meaningless, so Statedb was born, the library as the name implies is used to store state data, if there is statedb, the above scenario is the solution:

• Starting from the A account is the first revenue sys, to set up a table in Statedb for a account, to store the balance of the a account, each time the action of account A is transferred, it will update the value of the balance of the a account in the relevant table in Statedb, when we need to know the balance of a account , we can find this balance directly.

Test Cases

Here is a test method for everyone, and also my command history:

  Cleos Create Keycleos Wallet import 5ja9odotjhoknjuv6nramx4g5gwtcvcrlybtng1xvu3ekgzzenycleos Wallet Import --private-key 5ja9odotjhoknjuv6nramx4g5gwtcvcrlybtng1xvu3ekgzzenycleos Wallet Keyscleos Wallet Import--private-key 5kqwrpbwdl6phxujxw37fssqz1jiwsst4cqqzdeyxtp79zkvfd3cleos wallet Keyscleos Create account Eosio usertesta1 Eos761kfjhyy3fsypzgg5heprr8k2wzmw75jcdxgypkt2dlxzozpwcleos Create account Eosio Eosio.token Eos761kfjhyy3fsypzgg5heprr8k2wzmw75jcdxgypkt2dlxzozpwcleos Set Contract Eosio.token work/src/github.com/eos/build/ Contracts/eosio.token/cleos Push Action Eosio.token create ' [' Eosio ', ' 100000000.0000 SYS '] ' Cleos push action  Eosio.token issue Cleos Push Action Eosio.token issue ' ["Usertesta1", "10000000.0000 SYS"] '-p eosio.tokencleos push action  Eosio.token issue ' ["Usertesta1", "10000000.0000 SYS"] '-p eosiocleos get currency balance Eosio.token Usertesta1cleos get Table Eosio.token usertesta1 Accountscleos Get table Eosio.token eosio accounts  

You can see that when I want to get the balance of the USERTESTA1 account, I get it by querying the Statedb table, not the stupid way to sweep the block at the beginning.

The difference between chained storage and statedb storage

• Chained storage, which stores the fixed structure of data:block=> Block Header/transactions=>actions, an example of an action structure:

{    "account": "eosiotesta1",    "name": "hi",    "authorization": [{      "actor": "eosiotesta1",      "permission": "active"    }],    "data": {    "user": "yeah"    },    "hex_data": "0000000000d08cf2"}

In this example, we call the Hi function of the Hello contract, and the data is passed in as a custom in the Hi function, so in chained storage, the space left for us to play is here.

• Statedb, storage is a final record of the state, the state data must be meaningful, is someone concerned, insignificant data please do not put in statedb, so statedb can be deleted and modified, like a common database, in the contract through the Multi_ Index to operate, please refer to the article EOS Technology research: Contract and database interaction

Many people do not understand why the blockchain is not tamper-proof, but in the statedb as if can be modified can be deleted?

In fact, this is not the case, the content of the chain store will be all action action all recorded, is all the process data, is the flow account, metadata, these data once the chain is non-modifiable, can not be deleted. And Statedb is only to save a state information, this state information modification and deletion does not affect the block chain of the non-tamper characteristics.

At present, the mainstream implementation of Statedb is to put it in memory, and when some people on the understanding of statedb caused misuse, will cause memory overload, so on the one hand we need to understand very clearly the meaning of statedb, On the one hand Eosio helps us provide a mongodb-plugin plugin to synchronize statedb data.

MongoDB Installation

• Download TGZ installation package
• Unzip and install to/usr/local/bin (or some other path)
• sudo mkdir/data/db

Normal mode

• sudo mongod
• Mongo

Service mode

We can also use the Ubuntu system's service model.

Once we were going to define a system to start the service from the start of the way is to write a script in the/ETC/INIT.D directory to execute, now in Ubuntu service mode, we can discard that way, the service mode command is services, And now the Ubuntu system is advocating the use of the Systemctl command, the difference between their use is in the order of the parameters.

• Define a configuration file mongod.conf

• Define a service file to be placed in the/etc/systemd/system/

  sudo vi/etc/systemd/system/mongodb.service

[Unit]Description=High-performance, schema-free document-oriented databaseAfter=network.target [Service]User=mongodbExecStart=['mongod' command location] --quiet --config /etc/mongod.conf [Install]WantedBy=multi-user.target

• Find Service Status

  Systemctl List-unit-files

• Querying the activation status of a MongoDB service

  Systemctl is-enabled MongoDB

• Activating the system self-starting service

  sudo systemctl enable MongoDB

• Start MongoDB Service

  sudo systemctl start MongoDB

• Querying MongoDB Service Status

  sudo systemctl status MongoDB

• Stop MongoDB Service

  sudo systemctl stop MongoDB

Debug mode

IDE Select Clion,eos Source Download the latest, to ensure that the local can use the script compiled through, installed the relevant dependency package, and then import in Clion eos,clion will automatically identify CMakeList.txt file generation makefile file and make compile execution. You may encounter errors at compile time, in general either the environment dependency is not configured, or the CMakeList.txt to be modified, such as Mongodb-plugin on the general switch configuration when importing.

set(BUILD_MONGO_DB_PLUGIN "true")

After all the compilation is successful, the target that can be debug is automatically identified, corresponding to the module one by one with CMakeList.txt in Eos.

Installing the MONGO Explorer plugin

Above we introduced the MongoDB installation method, as well as the configuration method when starting Nodeos (in addition to the general switch mentioned above, of course, at the end of the config.ini file Set plugin = Eosio::mongo_db_plugin, this part of the content walkthrough several times, Don't repeat it here. When the chain starts to go out of the block, it syncs to MongoDB (note to pre-boot the Mongod daemon, which can be understood as the server side), MONGO command Access can use the MONGO command to query data, but this is inconvenient. You can install Mongo-plugin in Clion and configure the following effects: