BIGCHAINDB Source Analysis (iii)--back-end storage

BIGCHAINDB Source Analysis (i) Analysis of how bigchaindb parsing command-line parameters and configuration files, and accordingly started the log Publisher and Subscriber. For the bigchaindb Start command, the _run_init is invoked to initialize the back-end storage, followed by pipeline to initiate processes such as block\vote (BIGCHAINDB source Analysis (ii)). This section describes the Bigchaindb back-end storage.

_run_init is called by Run_start, and the code is located in commands/bigchaindb.py

Def _run_init ():
    # Try to access the KeyPair, throws a exception if it does not exist
    B = bigchaindb. Bigchain ()

    schema.init_database (connection=b.connection)

    b.create_genesis_block ()
    logger.info (' Genesis block created. ')

The function first creates a Bigchain instance, the Bigchain class is located in core.py, and its __init__ function loads the configuration file from Bigchaindb.config, including the public private key of this node, the public key of other nodes in the federation, the consistency algorithm plug-in, And to locate the classes that connect to the database, and so on. The database is then initialized based on the class that connects the database. Finally create the Genesis block. Create a Bigchain instance

The Bigchain class __init__ function joins the database code as shown below. Backend.connect is invoked when the connect parameter for class instantiation is none.

Self.connection = Connection If connection else Backend.connect (**bigchaindb.config[' database ')

The * * Before its formal parameter corresponds to the key value one by one of the dictionary in the formal parameter to the list of function arguments. For example:

>>> def Test (A, B, c):
...     Print (A, B, c)
...     
>>> Test (1,2,3)
1 2 3
>>> x= (1,2,3)
>>> Test (*x)
1 2 3
>>> y={ ' A ': 1, ' B ': 2, ' C ': 3}
>>> Test (**y)
1 2 3

The Backend.connect parameter list is as follows. Therefore, when invoked, the value one by one under database in the configuration file is mapped to the argument list.

# parameter list
def connect (Backend=none, Host=none, Port=none, Name=none, Max_tries=none,
            Connection_timeout=none, Replicaset=none, Ssl=none, Login=none, Password=none, Ca_cert=none, Certfile=none, Keyfile=none, Keyfile_
            Passphrase=none,
            Crlfile=none):

# Database Item
"Database" in configuration file: {
    "password": null,
    " Connection_timeout ": 5000,
    SSL": false,
    "Replicaset": "Bigchain-rs",
    "login": null,
    "Max_ Tries ": 3,
    " name ":" Bigchain ",
    " port ": 27017,
    " host ":" localhost ",
    " backend ":" MongoDB "
},

The Bigchaindb back-end store code is located in the backend directory and currently contains the MONGODB\RETHINKDB two types of back-end storage databases. Two back-end storage currently supported is also specified in the backend/connect.py.

backends = {
    ' MongoDB ': ' bigchaindb.backend.mongodb.connection.MongoDBConnection ',
    ' rethinkdb ': ' Bigchaindb.backend.rethinkdb.connection.RethinkDBConnection '
}

Backend.connect to find the back-end storage class code can be divided into the following steps: First is to get the value of the backend in the configuration file, then find the corresponding class from the variable backends, and then through the reflection mechanism of the class to load the string path

# Backend for MongoDB
backend = backend or bigchaindb.config[' database ' [' backend ']

module_name, _, class_name = Backends[backend].rpartition ('. ')
class = GetAttr (Import_module (module_name), class_name) return

class (Host=host, Port=port, Dbname=dbname,
     Max_tries=max_tries, Connection_timeout=connection_timeout,
     replicaset=replicaset, Ssl=ssl, Login=login, Password=password,
     Ca_cert=ca_cert, Certfile=certfile, Keyfile=keyfile,
     Keyfile_passphrase=keyfile_ Passphrase, Crlfile=crlfile)

Rpartition will return a triple, the string to the left of the last delimiter, the delimiter itself, and the string to the right of the last delimiter. Therefore, Class_name will correspond to mongodbconnection. And then using the GetAttr function to load the class, the variable class corresponds to an instance of the Mongodbconnection class, and finally calls the class class.

>>> x= ' bigchaindb.backend.mongodb.connection.MongoDBConnection '
>>> print (x.rpartition ('. '))
(' Bigchaindb.backend.mongodb.connection ', '. ', ' mongodbconnection ')

Let's look at the Mongodbconnection class for the Bigchaindb.backend.mongodb.connection module. The class inherits the Backend.connection.Connection class. In fact, all back-end storage connection classes should inherit the connection class and then implement their own code for manipulating the database. The __init__ function of the Mongodbconnection class will invoke the initialization function of the parent class, and then save some of the MongoDB unique configurations as members of the class, such as Replicaset. The __init__ of the parent class saves the configuration shared by some back-end storage as a member of the class, such as Host\port\dbname, and so on.

Class Mongodbconnection (Connection):

    def __init__ (self, replicaset=none, Ssl=none, Login=none, Password=none,
                 Ca_cert=none, Certfile=none, Keyfile=none,
                 Keyfile_passphrase=none, Crlfile=none, **kwargs):
        super (). __init__ (**kwargs)
        self.replicaset = Replicaset or bigchaindb.config[' database ' [' Replicaset ']

Initializing the database

The _run_init function initializes the database after the Bigchain instance is created. According to the previous analysis, the connection in the formal parameter is actually an instance of the Mongodbconnection class.

Schema.init_database (connection=b.connection)

# backend/schema.py
def init_database (Connection=none, Dbname=none):

    connection = connection or connect ()
    dbname = dbname or bigchaindb.config[' database ' [' name ']

    create_database (Connection, dbname) create_tables (connection,
    dbname)
    create_indexes (connection, dbname)

As you can see from the function name, the Init_database step is to create the database, table, and index sequentially. However, in backend/schema.py, these three functions actually have no function body, but there is an adorner singledispatch

@singledispatch
def create_database (Connection, dbname):
    raise Notimplementederror

@singledispatch
def create_tables (Connection, dbname):
    raise Notimplementederror


@singledispatch
def create_ Indexes (connection, dbname):
    raise Notimplementederror

The adorner Singledispatch the document in Https://pypi.python.org/pypi/singledispatch. By reading these examples, we can see that the function polymorphism is realized, that is, different functions of the same function name can be called by different parameters. The examples in the document are as follows: definition: Define a function that requires polymorphism through an adorner singledispatch, or register by @fun.register (int) or fun.register

>>> from Singledispatch import singledispatch
>>> @singledispatch
... def fun (ARG, verbose= False):     .. If verbose: ...         Print ("Let me just say,", end= "")
...     Print (ARG)

>>> @fun. Register (int)
... def _ (ARG, verbose=false):
...     If verbose: ...         Print ("Strength in numbers, eh?", end= "")
...     Print (ARG)

>>> def nothing (ARG, verbose=false):
...     Print ("nothing.")
...
>>> Fun.register (None), nothing

Call

>>> Fun ("Hello, World.")
Hello, world.
>>> Fun ("Test.", verbose=true) let
me just say, test.
>>> Fun (verbose=true)
strength in numbers, eh?
>>> Fun (None)
.

Thus, we look for functions that have been registered with Create_database in the back-end store. The following code is available in the MongoDB back-end storage Backend/mongodb/schema:

From bigchaindb.backend.utils import module_dispatch_registrar

Register_schema = Module_dispatch_registrar ( Backend.schema)

@register_schema (mongodbconnection)
def create_database (Conn, dbname):
    if dbname in Conn.conn.database_names ():
        raise exceptions. Databasealreadyexists (' database ' {} ' already exists '
                                               . Format (dbname))

    logger.info (' Create Database '%s '. ') dbname)
    # Todo:read and write concerns can is declared here
    conn.conn.get_database (dbname)

The implementation code for the Module_dispatch_registrar is:

def module_dispatch_registrar (module):
    def dispatch_wrapper (obj_type):
        def wrapper (func):
            func_name = func.__name__
            Try:
                Dispatch_registrar = getattr (module, func_name) return
                Dispatch_registrar.register ( Obj_type) (func)
            except Attributeerror as ex:
                raise Moduledispatchregistrationerror (
                    ' {module} ' does Not contain a single-dispatchable '
                     function named ' {func} '. The module being registered ' is not
                     implemented correctly! '). Format (
                         func=func_name, module=module.__name__)) from ex-return
        wrapper return
    Dispatch_wrapper

Look so complicated. According to the idea of the previous adorner, let's start with the replacement, first expand the Module_dispatch_registrar, the code is equivalent

@module_dispatch_registrar (Backend.schema) (mongodbconnection)
def create_database (Conn, dbname):
    ...

Also namely

@dispatch_wrapper (mongodbconnection)
def create_database (Conn, dbname):
    ...

# where variable module is Backend.schema
def dispatch_wrapper (obj_type):
    def wrapper (func):
        ...
    Return wrapper

and spread the dispatch_wrapper.

Wrapper (Create_database)

# where variable module is backend.schema, variable obj_type to mongodbconnection
def wrapper (func):
    ...
Return wrapper

So this code is actually called (Note that here all the variables are written in a string and actually refer to the corresponding object)

Dispatch_registrar = GetAttr (Backend.schema, Create_database)
Dispatch_registrar.register (mongodbconnection) (create_database)

Therefore, when Init_database executes code create_database (connection, dbname), the function called is connection because the Mongodbconnection type is the Backend/class. The Create_database function in mongodb/schema.py. The same is true of Create_tables and create_indexes. Creating a database

Thus, the function called by the creation database is backend.mongodb.schema.create_database, and the formal parameter conn corresponds to an instance of the Mongodbconnection class

@register_schema (mongodbconnection)
def create_database (Conn, dbname):
    if dbname in Conn.conn.database_ Names ():
        raise exceptions. Databasealreadyexists (' database ' {} ' already exists '
                                               . Format (dbname))

    logger.info (' Create Database '%s '. ') dbname)
    # Todo:read and write concerns can is declared here
    conn.conn.get_database (dbname)

Conn.conn is actually a property of the parent class of the Mongodbconnection class connection, because Self._conn value is none when the class __init__, so the Self.connect () is invoked. And connect will call the _connect function self.max_tries_counter times until the connection succeeds, or throws an exception

@property
def conn (self):
    if Self._conn is None:
        self.connect () return
    self._conn

def Connect ( Self):
    attempt = 0 for
    i in Self.max_tries_counter:
        attempt + = 1
        try:
            self._conn = Self._connect () C10/>except Connectionerror as exc:
            ...
        else: Break
            

The _connect function is shown below when the argument list is removed from exception handling and function calls. The function logic is also clear, first of all, to initialize the replica set, then use Pymongo to connect to the MongoDB database, then authenticate the account in the configuration file with the password, and finally return the connection and verify the successful client object. Note that the last client object returned will be assigned to the Conn member property of the Mongodbconnection class.

def _connect (self):
    try:
        initialize_replica_set (self.host,...)

        If Self.ca_cert is None or self.certfile are none or \
                Self.crlfile is none:
            client = Pymon Go. Mongoclient (self.host,...)
            If Self.login are not none and Self.password are not none:
                client[self.dbname].authenticate (Self.login, Self.password)
        Else:
            logger.info (' Connecting to MongoDB over Tls/ssl ... ')
            client = Pymongo. Mongoclient (self.host,...)
            If Self.login is not None:
                client[self.dbname].authenticate (self.login,
                                                 mechanism= ' mongodb-x509 ')

        Return client

    except ...

The Initialize_replica_set function is also the first to connect to the MongoDB database, and then check that the database has been configured with the replica set, and match the name of the replica set in the configuration file, if the match succeeds, the replica set has already been initialized, otherwise it is initialized. Create a table and create an index

The next step in creating a database is to create a table, which is simpler: Call the Pymongo create_collection function directly and create four tables under the database: Bigchain, backlog, votes, assets

@register_schema (mongodbconnection)
def create_tables (Conn, dbname):
    for table_name in [' Bigchain ', ' Backlog ', ' votes ', ' assets ']:
        logger.info (' Create '%s ' table. ', table_name)
        # Create the table
        # Todo:read A ND write concerns can is declared here
        conn.conn[dbname].create_collection (table_name)

After the table is successfully created, BIGCHAINDB creates a level two index for each of the four tables to facilitate the query, such as creating an index named transaction_id to support querying the transaction ID

@register_schema (mongodbconnection)
def create_indexes (Conn, dbname):
    Create_bigchain_secondary_index ( Conn, dbname)
    Create_backlog_secondary_index (conn, dbname)
    Create_votes_secondary_index (conn, dbname)
    Create_assets_secondary_index (conn, dbname)

def create_bigchain_secondary_index (Conn, dbname):
    Logger.info (' Create ' bigchain ' secondary Index. ')

    #
    to order blocks by timestamp conn.conn[dbname][' Bigchain '].create_index ([(' Block.timestamp ',
                                                 Ascending)],
                                               name= ' Block_timestamp ')

    # to query the Bigchain for a transaction ID, this field is unique
    conn.conn[dbname][' Bigchain '].create_index (' block. Transactions.id ',
                                               name= ' transaction_id ')
    ...

Now that the database is initialized, the only thing in the _run_init that hasn't been read is how to create the Genesis block, which is the next time you learn.