The distributed version of Atlas is hitting the ground! (1)

The distributed version of Atlas is hitting the ground! (1)

Atlas is a MySQL-based Data Middle Layer project developed and maintained by the infrastructure team of Qihoo 360 Web platform department. Based on MySQL-Proxy 0.8.2 officially launched by MySQL, it has modified a large number of bugs and added many functional features. Currently, this project is widely used within 360 companies. Many MySQL businesses have been connected to the Atlas platform, and billions of read/write requests are carried every day. At the same time, more than 50 companies have deployed Atlas in the production environment, and more than 800 people have joined our developer exchange group, and these numbers are constantly increasing.

Main functions:

1. read/write splitting

2. Slave database load balancing

3. IP address filtering

4. Automatic table sharding

5. DBA can smoothly bring online and offline DB

6. automatically remove the down Database

Overview of Atlas Sharding

Atlas Sharding is a key feature recently developed by Atlas. This feature increases the horizontal scalability and capacity of Mysql to meet the needs of most enterprises. It has been released on github with the Sharding branch.

The basic idea of Sharding is to split the data in a data table into multiple parts and store them on different hosts (there are multiple Sharding policies ), this reduces the performance and capacity of a single machine. sharding is a horizontal split that is suitable for scenarios with large data volumes in a single table. currently, atlas supports static sharding solutions and does not support automatic data migration.

Atlas shares sharding tables and non-sharding tables in the same database. Table data without sharding exists in a database group without sharding.

Currently, Atlas sharding supports insert, delete, select, and update statements, and does not support transactions across shard.

Of course, due to the limitations of Mysql's distributed architecture, Atlas Sharding has limited support for SQL features, but it is sufficient to meet daily requirements.

Differences from Mysql replication

MySQL Master-slave replication copies data from one MySQL instance (Master) to another MySQL instance (slave) in real time. This replication is an asynchronous replication process.

Data Replication has the following features:

• Data Distribution
• Load Balancing (using Atlas or other proxy Middleware)
• Backup
• High availability and fault tolerance

Replication has obvious limitations. When the database is frequently written but has few read operations, replication is not suitable. When writing is too frequent and it is difficult to be supported by a host, we are still faced with expansion bottlenecks. In other words, replication can only expand the read performance, but it is powerless to expand the write performance.

Sharding: based on a specific condition, data stored in the same database is distributed to multiple databases (hosts, to distribute the load of a single device. In this way, I/O is shared by shard when writing data. at the same time, replication exists on each Shard. With Atlas, read splitting can still be performed on Shard. Therefore, replication and Sharding are completely complementary and not excluded.

Sharding Architecture

Atlas is stateless. You can configure any number of Atlas instances for multiple backend groups. This is similar to MongoDB mongos.

Sharding database Group

In Atlas, a group is regarded as the unit of data storage. A group consists of one master, zero or multiple slave instances (the user must configure the mysql master-slave synchronization ). data in each group is independent and unrelated. Each part of the table data is stored in each group.

Intra-group read/write splitting

Atlas sharding also supports read/write splitting within the Group. That is to say, after a group is hit by Atlas, it still performs read/write splitting on the master and slave in the group (read to slave and write to master ).

Sharding data splitting Policy

Shard key

Each shard table has a shard key, which can be a primary key or a non-primary key, but this column must be an integer. atlas uses this shard key to determine which database group the record should be stored in.

Currently, Atlas Shardingh supports data sharding in the Range and Hash modes.