Data segmentation-Atlas introduction and segmentation atlas Introduction

Data segmentation-Atlas introduction and segmentation atlas Introduction

Atlas is a MySQL-based Data Middle Layer project developed and maintained by the infrastructure team of Qihoo 360 Web platform department. Based on MySQL-Proxy 0.8.2 officially launched by MySQL, it has modified a large number of bugs and added many functional features. Currently, this project is widely used within 360 companies. Many MySQL businesses have been connected to the Atlas platform, and billions of read/write requests are carried every day. At the same time, more than 50 companies have deployed Atlas in the production environment, and more than 800 people have joined our developer exchange group, and these numbers are constantly increasing.

Atlas Name:

Atlas: the giant with shoulders in Greek mythology, the brother of Prometheus, one of the largest and strongest gods, was fined for failing to resist Zeus. Qihoo 360 expects this system to be able to keep its feet on the back-end DB, thus setting up a sky for front-end applications.

Why Atlas?

Mysql-proxy is an official mysql middleware product that can achieve load balancing, read/write splitting, and failover. However, it does not support database/table sharding with large data volumes and has poor performance.

Advantages of Atlas over official MySQL-Proxy

1. rewrite all Lua code in mainstream processes with C, and Lua is only used to manage interfaces.

2. Rewrite the network model and thread model

3. implemented a true connection pool

4. optimized the lock mechanism and improved the performance by dozens of times.

Atlas Architecture Design:

Atlas is a middleware between applications and MySQL. In the view of backend DB, Atlas is equivalent to connecting to its client. In the view of front-end applications, Atlas is equivalent to a DB. As the server and Application Communication, Atlas implements the MySQL client and server protocols and communicates with MySQL as the client. It shields the database details from the application, and maintains the connection pool to reduce the MySQL burden.

Features and FAQs of Atlas:

1. Does Atlas support multiple character sets?

The support for multiple character sets is our first improvement to the original MySQL-Proxy and is required in line with national conditions. The client can specify the default character set during connection.

2. Does Atlas support transaction operations?

When a client in the transaction state exits midway through, Atlas will destroy the connection used by the client, so that mysql in the background can roll back the transaction, ensuring the integrity of the transaction.

3. Automatic read/write splitting is good, but sometimes I want to read it right away after writing it. What if the Master/Slave synchronization latency occurs?

Add/* master */before the SQL statement to forcibly send read requests to the master database. To test this function in the mysql command line, add the-c option to prevent the mysql client from filtering out comments.

4. Is the master database down and read operations affected?

Read operations in Atlas are not affected. Atlas forwards read requests to other surviving slave databases. However, the write request will fail because the master database is down.

5. Will detecting the backend DB status block normal requests?

No. The detection thread in atlas performs asynchronous detection. Even if a database goes down, the main process is not blocked. There are no exceptions in Atlas that will block the main process.

6. What should I do if I want to deprecate a database and stop mysql server?

You can use the management interface to manually connect backend databases to and from the backend databases. Atlas gives priority to the Administrator's wishes.

7. Do you want to add a DB to the cluster and do not want to affect normal online access?

It can be easily implemented through the management interface.

8. What are the improvements to Atlas compared to the official mysql-proxy?

A: This is really A hard-to-answer question. We have made dozens of improvements in terms of performance, stability, reliability, and maintainability. We will try to list some major changes below.

9. Does Atlas support the mysql prepare feature?

Currently, Atlas supports the prepare function and java, python, and PHP (PDO mode ).

10. Does Altas support the running mode of multiple master databases?

We have not tested the case that multiple master databases are attached to Atlas. We do not recommend this case. One master, one slave, or one master, multiple slaves are recommended.

11. When using Atlas, I found bugs in Atlas or had new functional requirements for Atlas. How can I provide feedback to developers?

We pay great attention to bugs reported by users. You are welcome to send an email to zhuchao [AT] 360.cnfor the environment, steps, and running of the recurring bug. In addition, if you have new functional requirements for Atlas in actual application scenarios, you can also send an email to us and we will reply promptly. In addition, enthusiastic netizens have created the QQ Group 326544838, and developers have joined the group to facilitate discussion.

12. java program connection to Atlas garbled

Delete the amp; In the jdbc connection, for example

Jdbc: mysql: // 10.10.10.37: 3306/user_db? UseUnicode = true & amp; characterEncoding = UTF-8 & amp; autoReconnect = true
To:

Jdbc: mysql: // 10.10.10.37: 3306/user_db? UseUnicode = true & characterEncoding = UTF-8 & autoReconnect = true

13. Monitor the latency between master and slave synchronization?

Atlas is not responsible for the master-slave synchronization of MySQL and must be managed by the DBA.

14. The java program connects to Atlas and finds that read/write splitting is not allowed. All requests are sent to the master database. Why?

Check whether the java framework sets autocommit to 0 by default. Many java frameworks encapsulate statements in a transaction, and Atlas sends transaction requests to the master database.

15. Does Atlas support the SQL statement blacklist filtering mechanism?

Yes, Atlas will block delete and update operations without the where condition, as well as sleep functions.