C ++ fflib ffcount: Universal Data Analysis System

Abstract:

Data analysis has become indispensable, and almost every company relies on data analysis for decision-making. In my online gaming field, data analysis is one of the most important means to plan new functions and optimize the gaming experience. Online Game data analysis has the following features (from the development perspective ):

• Large data volume; large online game users, large user behavior, large storage data volume.
• Real-time requirements are high. For example, new game functions, player experience, and feedback are expected to be analyzed as soon as possible.
• Demand changes rapidly. Online gaming demands are changing with each passing day, so the data analysis system is required to respond quickly to changes in demand.

Common Data Analysis Systems

Data analysis systems should be divided into data storage and data analysis. Common Data Analysis architectures include:

• Data analysis is customized directly in the logic service. In this case, MySQL or MongoDB is often used as the data storage. The advantage is that the customized data storage saves more space, the disadvantage is that the data storage servers of MySQL and MongoDB are often at a higher cost. If new requirements are added, customization requires a great deal of development, and old data analysis is maintained.CodeIt is often very difficult because it is highly customized and often bound to a special application background.
• Use scribe for data storage and hadoop for data analysis. Facw.k scribe server can use the hadoop distributed file system to store big data. E-commerce or SNS websites often use this scalable and mature solution. The disadvantage is that the deployment and maintenance costs are high, it is difficult for small and medium-sized teams to establish hadoop clusters, regardless of manpower or material resources.

Determine requirements:

• Data storage is as simple and cost-effective as possible. Because the reading efficiency of log data is not high, you can use general machines to store data on disks instead of using MySQL and other nosql.
• Data analysis is as simple as possible and easy to develop. Currently, SQL query is the most convenient and basic method, so data should be structured in SQL.
• Hadoop deployment is still daunting for Small and Medium-sized teams. Therefore, it is required that data analysis system deployment be simple and easy to configure.

Ffcount Architecture

 

Internal Working Mechanism

The sequence diagram shows the internal working mechanism:

 

Sample C ++ client code:

 

# Include "  Count/ffcount. h  "  # Include  "  RPC/broker_application.h  "  # Include  " Base/daemon_tool.h  "  # Include  "  Base/arg_helper.h  "  Using   Namespace  Ff; # include <Stdio. h> # Define Num 0 Int Main ( Int Argc, Char * Argv []) {arg_helper_t arg_helper (argc, argv ); If ( False = Arg_helper.is_enable_option ( "  -L  "  ) {Printf (  "  Usage: app-l TCP: // 127.0.0.1: 10241 \ n  "  );  Return   1  ;} Assert (  0 = Singleton_t <msg_bus_t>: instance (). Open (arg_helper.get_option_value ("  -L  " ))&& "  Can't connnect to broker  "  ); Assert (singleton_t <Msg_bus_t>: instance (). get_service_group ( "  Event_log_service  " )&& "  Event_log_service group not exist  "  ); Assert (singleton_t <Msg_bus_t>: instance (). get_service_group ( "  Event_log_service  " )-> Get_service ( 0 )&& "  Event_log_service 0 not exist  "  ); Event_log_t El (  "  Test  "  /*  Dbname  */ , " Dumy  "  /*  Tablename  */ , "  A, B, c  "  /*  Fields name  */ ); El. Def ( 100 , "  P \ "t' P  " , 5.4 ); Singleton_t <Msg_bus_t>: instance (). get_service_group ( "  Event_log_service  " )-> Get_service ( 0 )-> Async_call (EL );  For ( Int I = 0 ; I <num; ++ I ){  Char Buff [ 64  ]; Snprintf (buff, Sizeof (Buff ), "  Dumy _ % d  " , I % 8  ); Event_log_t El (buff,  "  A, B, c  " ); El. Def ( 100 , "  PP  " , 5.4  ); Singleton_t <Msg_bus_t>: instance (). get_service_group ( "  Event_log_service  " )-> Get_service ( 0 )-> Async_call (EL);} event_queryt_t: in_t in_msg; in_msg.db_name = "  Test  "  ; In_msg. SQL = "  Select * From dumy  "  ; Struct  Lambda_t {  Static   Void Callback (event_queryt_t: out_t & MSG _) {printf (  "  ====>>>>> Callback dump data [% s] <<<<<=======\ n  "  , MSG _. err_msg.c_str (); ffdb_t: dump (MSG _. ret_data, MSG _. col_names); event_log_t El (  "  Test  " ,"  Dumy  " , "  A, B, c  " ); El. Def ( 100 , "  P \ "t' P  " , 5.4  ); Singleton_t <Msg_bus_t>: instance (). get_service_group ( "  Event_log_service  " )-> Get_service ( 0 )-> Async_call (EL );  For ( Int I = 0 ; I <num; ++ I ){  Char Buff [ 64  ]; Snprintf (buff,  Sizeof (Buff ), "  Dumy _ % d  " , I %8  ); Event_log_t El (buff,  "  A, B, c  " ); El. Def ( 100 , "  PP  " , 5.4  ); Singleton_t <Msg_bus_t>: instance (). get_service_group ( "  Event_log_service  " )-> Get_service (0 )-> Async_call (EL);} Sleep (  1  ); Event_queryt_t: in_t in_msg;  //  In_msg.str_time = "2013/2 ";  //  ! Query the data of January 1, January In_msg.db_name = "  Test  "  ; In_msg. SQL = " Select * From dumy order by logtime DESC limit 5  "  ; Singleton_t <Msg_bus_t>: instance (). get_service_group ( "  Event_log_service  " )-> Get_service ( 0 )-> Async_call (in_msg ,& Lambda_t: callback) ;}}; singleton_t <Msg_bus_t>: instance (). get_service_group ( "  Event_log_service  " )-> Get_service (0 )-> Async_call (in_msg ,& Lambda_t: callback); signal_helper_t: Wait (); singleton_t <Msg_bus_t> : Instance (). Close ();  Return   0  ;} 

 

Example PHP Client

<? Phpfunction ffcount_query ($ host, $ port, $ str_time, $ db_name, $ SQL ){  //  Reference content is as follows:  //  1. Initialization $ CH =Curl_init ();  //  2. Set options, including URL $ Url = "  Http ://  " . $ Host. "  :  " . $ Port. "  /  " . $ Str_time. "  /  " . $ Db_name. " /  "  . Rawurlencode ($ SQL );  //  Echo $ URL. "\ n ";  Curl_setopt ($ ch, curlopt_url, $ URL); curl_setopt ($ ch, curlopt_returntransfer,  1  ); Curl_setopt ($ ch, curlopt_header,  0  );  //  3. Execute and obtain HTML document content $ Output = Curl_exec ($ ch );  // 4. Release the curl handle  Curl_close ($ ch );  If ($ Output = False ){  //  Echo "curl error:". curl_error ($ ch ); $ Ret = array ( "  Err_msg  " => "  HTTP request failed by curl  " , "  Col_names " => Array (), "  Ret_data  " => Array ());}  Else  {$ RET = Json_decode ($ output );  If (! $ RET) {$ RET = Array ( "  Err_msg  " => $ Output, " Col_names  " => Array (), "  Ret_data  " => Array ());}}  Return  $ Ret;} $ host = "  127.0.0.1  "  ; $ Port = 8080  ; $ Str_time = " 2013/2  "  ; $ Db_name = "  Test  "  ; $ SQL = "  Select * From dumy  "  ; $ RET = Ffcount_query ($ host, $ port, $ str_time, $ db_name, $ SQL); print_r ($ RET ); ?>

 

 Example C ++ server startup:

./App_count-l TCP: // 127.0.0.1: 10241-http tcp: // 127.0.0.1: 8080

Summary:

• Ffcount basically provides data log storage
• Ffcount uses SQL to organize log files and thus has the SQL data analysis capability.
• Ffcount data files are archived every month
• Ffcount automatically creates tables and fields. The autoid and logtime fields are created by default. The former is the auto-incrementing primary key, and the latter is the timestamp type. The default value is the current time.
• Ffcount supports HTTP queries. The data storage interface already has the C ++ class library interface

Build Server:

Git cloneHttps://github.com/fanchy/fflib

CD fflib/example/book/count & make &./app_count-l TCP: // 127.0.0.1: 10241-http tcp: // 127.0.0.1: 8080

Build client:

CD fflib/example/book/count_client & make &./app_client-l TCP: // 127.0.0.1: 10241

PHP client:

CD fflib/example/book/count_client/PHP & PHP test. php