Some empirical data to explore online services

This article provides online chat services based on some empirical data

A typical scenario: student data, which is based on historical data calculations that set hive user behavior data every morning, pushes data via tools to HBase, and obtains user behavior data through the business partner RPC Service

Overall architecture

Watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvewzraxnz/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/dissolve/70 /gravity/southeast "width=" 376 "height=" >
Schedule: Start compute/Transfer tasks
Monitoring: Monitoring of Task Execution (visual interface). Abnormal alarms
Quality: Quality inspection of core data
Transfer: Import data from hdfs/hive to Hbase/redis/memcache/mysql
RPC service: Getting Data through service
JDBC: Get data from MySQL
Memcache: As a query buffer layer for Hbase/redis

Business Access Storage Select a new data service. When choosing a storage type, follow the following principles:
1. Priority to use Hbase/redis. Hbase/redis Unable to support query semantics using MySQL (paging query, Count)
2. Small data volume (memory consumption <10g level) or use Redis (user profile) for applications with very high query response time requirements
3. Use HBase for massive data (user behavior pipelining/messaging)

Transmission Design reference Ali Datax Design, the implementation of the review of the Heterogeneous data Offline transmission tool wormhole. Supports HBase, Redis, Memcache, MySQL write
After the offline data is calculated, the data is exported from hive to HBase, Redis, Memcache, Mysql via Wormhole
Problems encountered:
1. Data affects query performance at write time
Scheme:

• Incremental import: Implements a data-control tool on hive, importing only change data to the online service store
• Speed limit import: speed limit during conduction

2. How to ensure the data import process data availability
For MySQL: Write data to a temporary table before switching
For Hbase/memcache/redis: Write using Update. Does not affect data availability
3. What if the transfer task fails?
A Sqoop-tolerant solution written before the reference. The idea is basically the following:
For a transport tool/platform, the transmission task failure is not scary, and the scary part is how the "dirty data" is handled. 3 Ways of thinking:
1. Temporary tables: Use temporary tables to cache data and then move temporary table data to the destination table in a transaction
2. Define rollback yourself: After a task fails, through a user-defined statement/method. Run Clear Data operations
3. The power of the transfer task: Suppose a task fails, resulting in dirty data. After solving this problem, run a task again. Can finally be correct, such as hive write using Insert OVERWRITE

HBase Scenario Design hbase Cluster main goals include:
1. Support massive data (level billions of) fast access
2. Cluster high Availability
3. Finally consistency
Scheme:
1. Two sets of Hbaes clusters. M-M do replication
2. User access via service
3. Encapsulate the client. Main work: Ease of use, cluster switching
Demo sample, an HBase query API:

/** * Single query: Returns value by table name, key, family, column *  * @param tableName *            table name * @param key *            query key       * @param family *            the family (specification definition: "Dim") where the column resides * @param column *            column name * @return  *            Returns the value of the specified column, the  specified column does not exist return NULL * @throws  *            IOException Remote call failed or network exception */public string query (string tableName, String key, String family,string column) throws Ioex ception;

Experience and issues:
1. How to use two sets of clusters?

• Under normal circumstances, visit a cluster, assuming that the cluster fails, manually cut the request to a cluster
• When making cluster changes, the backup cluster assumes the responsibility of "pre-environment"
• Individual high-write applications and other applications are distributed across different clusters, in the case of a cluster problem. Cut to a cluster, service degraded (response time becomes larger)

2. HBase Encounters problems

• Not suitable for applications that require very tight response times
• Make changes to the line (configuration changes. expansion, etc.), the risk is very big, cautious! Careful.

3. Why request by RPC service

• Improved ease of Use
• Package Cluster Switching
• Jar package dependent coupling too high
• Hadoop Kerberos security enables high deployment costs and is not conducive to business expansion

Redis Cluster Solution we introduced hbase earlier than Redis for demand reasons, but HBase was unable to meet the fast read and write requirements for random data. Therefore, Redis is introduced. And did a cluster solution
Its design objectives:
1. Support fast Random read/write service
2. High Availability
3. Finally consistency
Solution:

Watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvewzraxnz/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/dissolve/70 /gravity/southeast "width=" 364 "height=" >

• Access via RPC service
• Client Masks multiple Reids Node
• The Sentine cluster is responsible for master-slave switching and reporting to the client which node is currently the master node.
• READ: Polling
• Write/delete: Redis Master
• Bulk Write/delete: Partial failure all failures

Redis Deployment

• The machine with 128G memory on the line, 8 Redis-server nodes per machine, each node memory usage is controlled within 10G, the goal is to reduce the impact of the Redis Servier Node Fork Sub-process
• Consider our application scenario. Turn off your active bgrewriteaof on line. The Redis-server node sequence on the same machine every morning is bgrewriteaof
• Monitoring: Zabbix+logscan

Memcache: Using the company's existing solutions
Encapsulates the client,client with a consistent hash, multiple memcache machines to form a cluster. Provides storage capabilities beyond the single machine memory
No guarantee of availability

Model in our application, more than 90% of requests are in KV type request
Typical applications:
Use UserID to query the user's sex and age
For this type of demand, we have unified the Hbase/memcache/redis query model for the goal of:
Data development students have added a new field to hive. The Business party line can request this field without knowing what the underlying storage is
Model:

TableName

Family

Key

Columns

Tablename:hive table name, including schema name, demo sample: Bi.dprpt_user_city_profile_service
Family: a column cluster. A column family can be made up of arbitrary multiple column
Key: Business key (e.g. Userid,guid)
Qualifiers in Columns:hbase, serialized as JSON in Redis/memcache

In HBase, the above concepts are consistent with HBase
In the Reids/memcache:
TableName + Family + key organization becomes key for physical storage
Columns is serialized as JSON
thereby unifying the query interface:

/** * Single query: Returns value by table name, key, family, column list *  * @param tableName *            table name * @param key *            query key       * @param fam ily *            column family (default: "Dim") * @param columnlist *            column list * @return  * Returns the            specified column (JSON format) * @throws  *            IOException Remote call failed or network exception */public string query (string tableName, String family, string key,list<string> columnlist) Throws IOException;

Demo sample, a table in hive:
HIVE > select * from Bi.dprpt_user_city_profile_service limit 1;
Ok
1_1 2010-09-02 2014-05-02 2010-09-02 2014-05-02 2011-06-01 2013-09-27 2010-06-08 20 13-09-16 2012-08-29 2013-07-08 2 101 842
Save in HBase as:
HBase (main):001:0> get ' bi.dprpt_user_city_profile_service ', ' 1_1 '
COLUMN CELL
Dim:first_app_tg_date timestamp=1400025620039, value=2012-08-29
Dim:first_app_visit_date timestamp=1400025620039, value=2010-09-02
Dim:first_tg_date timestamp=1400025620039, value=2010-06-08
Dim:first_tg_visit_date timestamp=1400025620039, value=2011-06-01
Dim:first_visit_date timestamp=1400025620039, value=2010-09-02
Dim:last_app_tg_date timestamp=1400025620039, value=2013-07-08
Dim:last_app_visit_date timestamp=1400025620039, value=2014-05-02
Dim:last_tg_date timestamp=1400025620039, value=2013-09-16
Dim:last_tg_visit_date timestamp=1400025620039, value=2013-09-27
Dim:last_visit_date timestamp=1400025620039, value=2014-05-02
Dim:prefer_tg_cat0 timestamp=1400025620039, value=2
DIM:PREFER_TG_CAT1 timestamp=1400025620039, value=101
Dim:prefer_tg_region timestamp=1400025620039, value=842

Save in Redis/memcache as (the value after 7036447591228069586 is MurmurHash for Tablename+family+key):
Redis > Get 7036447591228069586
"{" Prefer_tg_cat0 ":" 2 "," Last_tg_date ":" 2013-09-16 "," last_tg_visit_date ":" 2013-09-27 "," Last_app_visit_date ":" 2014-05-02 "," first_app_tg_date ":" 2012-08-29 "," first_visit_date ":" 2010-09-02 "," last_app_tg_date ":" 2013-07-08 "," PREFER_TG_CAT1 ":" 101 "," Prefer_tg_region ":" 842 "," first_tg_date ":" 2010-06-08 "," last_visit_date ":" 2014-05-02 "," First_app_visit_date ":" 2010-09-02 "," first_tg_visit_date ":" 2011-06-01 "}"

Performance in our scenario: with a key, get multiple column
Individual storage response time (in ms):

	Avg	95- Wire	99.9-Wire
HBase	2.2	7.8	62.6
Memcache	0.7	1.0	3.6
Redis	0.3	0.6	1.2

Cluster size, QPS, scene is not the same, will cause the above data is not the same, for reference only

Some empirical data to explore online services