System Overview of the Aerospike-architecture series (Systems Overview)

Systems Overview (System Overview)

Aerospike is a distributed, extensible NoSQL database built for three main goals:

• Create a resilient, extensible platform that meets today's web platform applications
• Provides the same robustness and reliability as traditional databases (for example, ACID)
• Provides operational efficiency (minimum manual participation)
  

first published on Proceedings of VLDB (Very Large Databases)in, Aerospike architecture includes three layers:

1. The cluster-aware client layer includes an open-source client function library that implements the Aerospike API, tracks nodes and senses where the data resides in the cluster
2. Self-managed cluster (clustering) and data distribution Layer monitor cluster communication and automatic failover, data replication, cross-datacenter synchronization, smart rebalancing, data migration
3. Flash-optimized data Storage reliably stores data in memory and flash
   

Client Layer

Aerospike "Smart client" is designed for speed. It is implemented as an open source, linked library for C, C #, Java, PHP, and Python development, and developers can freely publish or modify as needed. The client includes the following content

• Implement the Aerospike API with a direct connection to the cluster via client-server protocol
• Tracks node-aware data storage locations and immediately senses cluster configuration changes as nodes start or stop.
• The TCP/IP connection pool is implemented for increased efficiency. Detect non-cluster node failure level transaction failure and new route transaction to data copy
• Transparently send requests directly to the node's data and retry or re-request as needed, an example of which is during cluster reconfiguration.

This architecture reduces transaction latency, diverts cluster work, and eliminates developer workload. It ensures that the application does not have to be restarted at node start-stop. In summary, it eliminates the need for additional cluster Management servers or proxy servers.

Distribution Layer

The Aerospike "shared nothing" architecture is designed to reliably store terabytes and data and support automatic fault tolerance, replication, and synchronization across data centers. This layer achieves linear expansion and ACID guarantees. The distribution layer is also designed to eliminate manual operation and to automate all cluster management functions of the system. It consists of 3 modules:

• Cluster Management Module for tracks cluster nodes. The key algorithm is the Paxos-like consistent voting process that determines which nodes are part of the cluster. The aerospike enables dedicated heartbeat detection (active vs. passive) for monitoring connectivity between nodes.

• When a node is added or removed and the cluster members are determined, each node uses a hashing algorithm to divide the primary index space into data slices and assign its owner. data Migration Module ( Data Migration module" Then intelligently balances the data distribution across the nodes in the cluster and ensures that each data block Kua node and the Kua data center replicate according to the replication factor of the system configuration. Data segmentation is purely an algorithm, the system is extended without master, thus eliminating other additional configurations in a shared environment.

• The Transaction processing module (transaction processing module) is used to read and write data requests and guarantees consistency and isolation, which is responsible for
  
  1. Sync/async Replication (Synchronous/Asynchronous replication): to ensure write consistency, propagate updates to all replicas before committing the data and return the results to the client.
  2. Proxy : during cluster reconfiguration, the client may have a brief expiration and transparent proxy requests to other nodes.
  3. Duplicate Resolution (replica resolution): resolves conflicts between different copies of data when the cluster resumes from the active partition.
     
     • Automatic, with data at the latest timestamp as standard
     • User-driven, to determine the level of data, all copies of the data are returned to the application.

Clustering

Once the cluster is started, you can install other clusters of cross data-center replication in other datacenters , and if the cluster crashes, the remote cluster can take the load.

Data storage Tier

Aerospike stores key-value pairs in a modeless data model. The container for organizing data, called a namespace (namespaces), is equivalent to databases The concept of a library () in an RDBMS system. In namespace, the data is subdivided into collections (set) (tables in a database) and records ( records ) (similar to rows in a database). Each record in the set has a unique index key and one or more bins (columns in a database-like column) associated with it.

• Set and bin do not need to be pre-defined and can be added at run time.
• The values in the bin are strongly typed, including any supported data types. The bin itself is not a type, so the same bin name can be different types of data.

For quick access, the index (primary keys and secondary keys) is stored in memory and the data can be stored in memory or SSD drives. Each namespace can be configured separately so that small namespace can be stored in memory and the large namespace can be stored on the SSD.

The data layer is specifically designed to increase speed while reducing hardware costs. It can act as a buffer layer for all operations in memory or take advantage of optimized flash storage, which does not lose data.

• 100 million keys only occupy 6.4GB of space. Although key has no size limit, the valid storage for each key is only 64bytes
• Native, multi-threaded, multi-core flash I/O and aerospike log structure file systems utilize low-level SSD read and write modes. In addition, the write disk performs block write operations to reduce latency. This approach bypasses the standard file system to optimize traditional disks.
  
• inside Smart Defragmenter ( build Smart defragmentation) and Intelligent Evictor ( Smart Cleaner). These processes work together to ensure that in-memory data is not lost and securely written to disk
  
• • Defragmenter tracks activity records in each piece and reclaims those blocks below the lowest usage rate
    
  • The cleaner removes expired records and reclaims memory when the system reaches the high watermark. Expiration time configuration in each namespace, the record lifetime is calculated from the last modified time, and the app can reassign the expiration time over the default data lifecycle to set the data to never expire.
    

Operation Aerospike

In a traditional (non-distributed) database system, you need to set up the schema, create the database, and table the software. This differs greatly from the Aerospike database.

In a distributed database, data is distributed across servers in the cluster. This means that you cannot access all the data on a single server.

Use Aerospike database, the database is created and managed as follows:

• Initialize the database settings by configuration. In terms of aerospike, when installing a system a library is called a namespace, and each node in the cluster must indicate how each namespace is created and the number of replicas. The database is created when you restart the service
  

• Perform database operations through the application. When the application first refers to set and bin, the database schema is created, and the application simply stores the data in the specified bin. In the Aerospike database, tasks are typically performed by the DBA through a command-line program.
  

• Modify the configuration file as needed. To update the configuration parameters of namespace, you need to dynamically modify or restart the service using the new configuration file.
  

to meet performance and redundancy requirements, Aerospike needs to plan and configure the exact number of nodes, as detailed in Capacity Planning

Management utilities and monitoring Tools can be used to manage and monitor the nodes of the urgent masses. The cluster is automatically configured when nodes are added or when downtime is required for upgrade maintenance. When a node fails, the cluster balances the load to achieve the least impact on the end user.

Building the Application

Once created Namespace,aerospike provides a tool that allows you to verify database storage data correctness. In the production database, the data is distributed among the clusters. In order to manipulate the database you need to instantiate the smart client in the application. Smart terminals are location-aware and know how to store/retrieve data in a cluster without compromising performance.

Aerospike provides multiple language APIs for building big data applications. Refer to the client manual for details

When compiling an application, the API library is included with the smart terminal. To determine the location of the data at any given time, the intelligent terminal continuously monitors the cluster status. The smart client's location-aware technology ensures that, in most cases, the required data can be retrieved within a single hop.

When it comes to big data applications, such as Web-based applications, the scenario is as follows:

Smart clients allow applications to ignore data distribution details. Please refer to Architecture Guide for specific details.

in this document, we will use the termAPI andthe client interchangeably-integrated Aerospike API application integrates the smart terminal at the same time.

original link: >  

System Overview of the Aerospike-architecture series (Systems Overview)