Daily request million level QQ member AMS platform PHP7 Upgrade Practice

QQ member activity Operation Platform (AMS), is one of the important carriers of QQ member value-added Operation business, and undertakes the web system of mass activity operation. AMS is a major implementation of the PHP language activities operating platform, CGI Day request about 300 million, peak reached 800 million. However, in the previous long period, we have adopted the older version of the basic software, that is php5.2+apache2.0 (2008 technology). Especially since last year, with the AMS business with the rapid growth of QQ member value-added services, the performance pressure has become increasingly large.

As a result, since May 2015, we have started to plan PHP bottom-up upgrades, with the ultimate goal of upgrading to PHP7. At that time, PHP7 was still in the research and development phase, and our discussion and pre-research had already begun.

First, PHP7 study and pre-research 1. HHVM and JIT

2015 on the PHP performance optimization scheme, there is another important role, is the Facebook open source HHVM (HipHop virtual MACHINE,HHVM is a Facebook open source PHP virtual machine). HHVM uses JIT (Just in time, instant compilation is a software optimization technique that compiles bytecode to machine code at run time) and other techniques, allowing PHP code to perform significantly better. It is rumored that the PHP5 version of native PHP code can be improved by up to 5-10 times the execution performance.

HHVM originated from Facebook, and many of Facebook's early-morning code was developed using PHP, but with the rapid development of the business, the efficiency of PHP execution became more and more obvious. To optimize execution efficiency, Facebook started using hiphop in 2008, a PHP execution engine originally designed to turn Fackbook's massive PHP code into C + + to improve performance and conserve resources. PHP code that uses hiphop has several times the performance boost. Later, Facebook will open up the hiphop platform and evolve into the current HHVM.

When HHVM becomes a PHP performance optimization solution, PHP7 is still in the research and development phase. Have seen some of the students for HHVM communication, performance can be a considerable improvement, but service delivery and PHP syntax compatible with a certain cost. For a while, JIT became a very loud thing, many technical classmates suggest that PHP7 should also be JIT to optimize performance.

July 2015, I participated in the Chinese Phpcon, listened to Xinchen about the PHP7 core technology sharing. In fact, in 2013, Xinchen (PHP7 kernel developer) and Dmitry (one of the other PHP language kernel developers) had made a JIT attempt on the PHP5.5 version (not published). PHP5.5 's original execution process, is the PHP code through lexical and syntactic analysis, compiled into opcode bytecode (format and assembly a bit like), and then, Zend Engine read these opcode instructions, one by one parse execution.

They introduce type inference (typeinf) after the opcode link, and then generate Bytecodes by JIT and then execute.

Therefore, in the benchmark (test program) to get very good results, the implementation of JIT performance than PHP5.5 8 times times. However, when they put this optimization into the actual project WordPress (an open source blog project), the performance improvement was almost invisible. The reason is that the code of the test project is relatively small, the machine code generated by the JIT is not big, and the real WordPress project generated by the machine code is too large, causing a decrease in CPU cache Hit rate (CPU cache Miss).

In short, the JIT is not in every scenario is a gold-touch tool, and the performance test results from the business scenario is not necessarily representative.

From the official release WordPress PHP7 and HHVM Performance comparison can be seen, the two are basically at the same level.

2.PHP7 optimization in terms of performance

PHP7 is a relatively low-level upgrade, compared to the PHP5.6 change is relatively large, and on the performance optimization layer, can be summarized as follows:

(1) Change the underlying variable from struct (struct) to Union (union), save memory space, and indirectly reduce CPU overhead in memory allocation and management.

(2) Some basic variables (Zend_array, zend_string, etc.) adopt the method of continuous allocation of memory space to reduce the probability of CPU Cache miss occurring. CPUs get data from the CPU cache and get it from memory, which can be up to 100 times times more efficient. As an example, the efficiency of the system to read data from memory and read data from disk varies greatly, and CPU Cache miss is similar to encountering a missing fault.

(3) With macro definition and inline function (inline), let the compiler complete some work ahead of time. There is no need to allocate memory while the program is running, it can implement functions similar to functions, but there is no stack and stack overhead for function calls, which is more efficient.

... ...

More details about PHP7, interested students can view: "PHP7 Innovation and performance optimization"

Background of 3.AMS platform technology selection

To improve the performance of PHP, you can choose the 2015 can be directly used HHVM or the end of 2015 to release the official version of the PHP7. Member AMs is a large-scale access to a web system, after four years of continuous upgrade and optimization, accumulated more than 800 business functional components, as well as a variety of PHP written public base libraries and scripts, code size is also relatively large.

We have a high demand for the compatibility of the PHP version of the code, so, in terms of our business scenario, PHP7 good syntax backwards compatibility is exactly what we need. Therefore, we chose to PHP7 as an upgrade scenario.

Ii. risks and challenges faced by PHP7 upgrade

For an already online large-scale public Web services, the basic public software upgrade, usually a thankless job, do well, not necessarily be perceived by everyone, but, upgrade out of the problem, you need to bear the heavier responsibility. To minimize the risk of escalation, we must first understand the challenges and risks of our upgrade.

So we sorted out the upgrade challenge and risk list:

(1) Apache2.0 and PHP5.2 These two 2008-2009 years of basic software version is older, upgrade to Apache2.4 and PHP7, version upgrade span is relatively large, time span of 7-8 years, therefore, compatibility issues challenge is higher. In fact, our company's current web PHP services, many are stuck in PHP5.2 and PHP5.3 version, the version is low.

(2) AMs Extensive use of self-developed tphplib extension, Tphplib very early in the company has no one to maintain, this extension only PHP5.3 and PHP5.2 compiled so version, and, some extensions do not support thread safety. Thread safety is supported because our previous Apache used the prefork mode, And we want to be able to use Apache2.4 's event mode (2014, after prefork and workers, the introduction of multi-process thread management mode, to support high concurrency, better performance).

(3) The problem of grammatical compatibility, from PHP5.2 to PHP7 the span is too large, even if the official PHP is claimed to be 99% in terms of backward compatibility, but our code size is larger, it is still an unknown risk.

(4) The risk of new software, the basic software such as Apache and PHP to upgrade to the latest version, and these versions of some of the features may have unknown risks and defects.

Some students may suggest that the use of Nginx will be more excellent choice, indeed, simple comparison of Nginx and Apache in high concurrency performance, nginx performance is more excellent. But in terms of PHP CGI, NGINX+PHP-FTPM and apache+mod_php do not have a big gap. On the other hand, because of the long-term use of Apache, we have accumulated more knowledge and experience in technology, so it may not be the best choice, but, specifically to our business scenario, is a more appropriate choice.

Third, version upgrade implementation process 1. High-SPAN version upgrade mode

From a 2008-year Apache2.0 direct upgrade to the 2016 Apache2.4, this span is too large, even the use of http.conf configuration files are many different, here need to update more places, the unknown risk is also there. So, our approach is to first try to upgrade Apache2.0 to Apach2.2, adjust the configuration, observe the stability, and then further try to Apach2.4. Fortunately, Apache (httpd) is a relatively special open source community that has previously maintained the two-spoke versions of Apache (2.2 and 2.4), so even Apache2.2 has a newer version.

So we first upgraded a php5.2+apache2.2, tested and observed compatibility, and confirmed that there was a smoother upgrade between the two, and we started to Apache2.4 the upgrade scenario.

PHP5.2 upgrade, we also use the same idea, we first upgrade the PHP5.2 to PHP5.6 (at that time, PHP7 or beta version), and then upgrade PHP5.6 to PHP7, in a smoother way, gradually solve the different problems.

As a result, our upgrade plan becomes:

Apache2.4 is compiled into a dynamic MPM mode (supports switching prefork/worker/event mode via HTTPD configuration), and is degraded in real time based on the current network risk.

Prefork, Worker, event three rough introduction:

(1) Prefork, multi-process mode, 1 processes serving 1 user requests, high cost. However, the highest stability is not required to support thread safety.

(2) Worker, multi-process multithreaded mode, 1 processes contain multiple worker threads, and 1 worker threads serve 1 user requests because the threads are lighter and less expensive. However, in the keepalive scenario, the worker resource is occupied by the client and cannot respond to other requests (empty waits).

(3) event, multi-process multithreaded mode, 1 processes also contain multiple worker threads, and 1 worker threads serve 1 user requests. However, it solves the problem of worker threads being occupied in the KeepAlive scenario, which manages these keepalive connections through a dedicated thread, and then assigns "work" to the specific worker, and the worker does not cause an empty wait due to keepalive.

Official introduction to the event mode:

Http://httpd.apache.org/docs/2.4/mod/event.html

(Some students may have an event mode does not support the impression of HTTPS, the statement is actually more than 2 years ago, the domestic part of the technical blog, the current version is supported, details can be viewed official introduction)

The way to turn on dynamic switching mode is to add the following when compiling httpd:

–enable-mpms-shared=all

Upgrading from PHP5.2 to PHP5.6 is relatively easy, and our main work is as follows:

(1) Cleaning up old extensions that are no longer used

(2) Resolve thread safety issues

(3) Compiling APIs such as Cmem to a new version

(4) PHP code syntax is based on PHP5.6 compatibility (in fact not changing very much)

(5) Part of the extended synchronization adjustment. The APC extension becomes Zend_opcache and APCU, and the previous APC is a feature that contains the compile cache and user memory operations, which are broken down into two separate extensions in PHP compared to the new version.

The workload of upgrading from PHP5.6 to PHP7.0 is more and more complex, so we have developed an upgrade plan for each of these phases:

(1) Technical pre-research, PHP7 upgrade preparation.

(2) The environment compiles and builds, downloads the related compilation package, constructs the complete compilation environment and the test environment. (The compilation environment still needs to be more dependent so)

(3) compatible with upgrades and tests. PHP7 extended recompile and code compatibility work, AMS functional verification, performance pressure measurement.

(4) on-line grayscale. Package the installation package as PKG, write the relevant installation shell installation execution code (including soft links, resolve some so dependencies). Then, the grayscale installation to the current network, observation.

(5) Official release. Enlarge the grayscale range and upgrade the whole volume.

Because of the process of upgrading from PHP5.2 to PHP5.6, a lot of problems have been solved in advance, so the main difficulty of upgrading PHP7 is Tphplib extension of the compilation upgrade.

The main tasks involved include:

(1) PHP5.6 expansion to the PHP7.0 of the comparison of large-scale upgrading (the workload is relatively large place)

(2) The name of the memory operation function that is compatible with APCU. PHP5, the function of the APC prefix we used is not available, and the function of synchronizing to the APCU prefix (requires APCU extension).

(3) syntax compatibility upgrade. In fact, the workload is not large, from PHP5.6 upgrade to PHP7 change is not much.

We probably completed the compilation of PHP7 and Apache in mid-April 2016, and in late April we had a full-scale release to one of the current network clusters at the beginning of May.

2. Error debugging methods during the upgrade process

When upgrading and recompiling the PHP7 extension, if the execution results do not conform to expectations or the process core is dropped, many errors cannot be seen from the error log and are not conducive to analyzing the problem. You can use the following methods to locate and analyze most of the problems:

(1) Var_dump/exit

From the PHP code layer to gradually output information and execution exit, you can step into the exception of the PHP function execution location, and then based on the PHP function name, the extension of the implementation function to find the problem. This method is relatively simple, but not high efficiency.

(2) Gdb–p/gdb C

This method is mainly used to analyze the scene of the process core, we use the compilation method is to mod_php (PHP into Apache sub-or Block way), using GDB–P to monitor the Apache service process.

Command: PS aux|grep httpd

GDB debugs The specified process:

Command: Gdb-p

Use C for capture, and then construct a Web request that can cause core:

Apache is usually a multi-process mode, in order to make the problem easier to reproduce, you can modify the parameters in the Http.con, the number of start-up process is modified to 1 (multiple parameters in the need to adjust to achieve only the purpose of single-process startup).

There is, of course, a simpler approach, because Apache natively supports single-process debug mode.

./apachectl-k Start-x-E Debug

Then it's easier to debug with gdb–p.

(3) through the Strace command to see what the Apache process is doing, based on the contents of the implementation, analysis and positioning problems.

Strace-ttt-v-s1024-f-P PID (process ID)

Note: Execute these commands, pay attention to permissions issues, and most likely require root privileges.

Iv. PHP5.6 to PHP7.0 extended upgrade practice record 1. Data type changes (1) zval

The birth of PHP7 began with the change of zval structure, PHP7 no longer needed pointers, most zval** needed to be modified into zval*. If the PHP7 direct Operation Zval, then zval* also need to change to zval,z_*p () also to change to z_* (), zval_* (Var, ...) Need to change to zval_* (&var, ...), be careful to use the & symbol, because PHP7 almost does not require the use of zval*, so many places & is to be removed.

Alloc_zval,alloc_init_zval,make_std_zval these memory-allocated macros have been removed. In most cases, zval* should be modified to Zval, and Init_pzval macros will be removed.

/* 7.0zval structure source code */
/* value field, only one size_t length, only pointer or double or long */
Typedef union _zend_value {
     Zend_long lval; /* long value */
     Double dval; /* double value */
     Zend_refcounted *counted;
     Zend_string *str;
     Zend_array *arr;
     Zend_object *obj;
     Zend_resource *res;
     Zend_reference *ref;
     Zend_ast_ref *ast;
     Zval *zv;
     Void *ptr;
     Zend_class_entry *ce;
     Zend_function *func;
     Struct {
         Uint32_t w1;
         Uint32_t w2;
     } ww;
} zend_value;

Struct _zval_struct {
     Zend_value value; /* value */
     Union {
         . . .
     } u1;/* Augment field, mainly type information */
     Union {
         ...
     } u2;/* Augment field to save auxiliary information */
};

(2) Integral type

You can switch directly:

Long->zend_long

/* Definition */
Typedef int64_t zend_long;
/* else */
Typedef int32_t zend_long;

(3) String type

The PHP5.6 version uses char* + Len to represent the string, PHP7.0 is encapsulated and defines the zend_string type:

struct _zend_string {
    zend_refcounted_h gc;
    zend_ulong        h;                /* hash value */
    size_t            len;
    char              val[1];
};

Conversion of zend_string and char*:

Zend_string *str;
Char *cstr = NULL;
Size_t slen = 0;
//...
/* Get char* and len from zend_string as follows */
Cstr = ZSTR_VAL(str);
Slen = ZSTR_LEN(str);
/* char* Constructs a method of zend_string */
Zend_string * zstr = zend_string_init("test",sizeof("test"), 0);

Extension methods, when parsing parameters, use a string where the ' s ' is replaced with ' s ':

/* E.g */
Zend_string *zstr;
If (zend_parse_parameters(ZEND_NUM_ARGS() , "S", &zstr) == FAILURE)
{
     RETURN_LONG(-1);
}

(4) Custom objects

Source:

/* php7.0 zend_object definition */
Struct _zend_object {
     Zend_refcounted_h gc;
     Uint32_t handle;
     Zend_class_entry *ce;
     Const zend_object_handlers *handlers;
     HashTable *properties;
     Zval properties_table[1];
};
/* example */
Struct clogger_object {
     CLogger *logger;
     Zend_object std;// placed behind
};
/* Get the object using the offset */
Static inline clogger_object *php_clogger_object_from_obj(zend_object *obj) {
     Return (clogger_object*)((char*)(obj) - XtOffsetOf(clogger_object, std));
}
#define Z_USEROBJ_P(zv) php_clogger_object_from_obj(Z_OBJ_P((zv)))
/* When releasing resources */
Void tphp_clogger_free_storage(zend_object *object TSRMLS_DC)
{
     Clogger_object *intern = php_clogger_object_from_obj(object);
     If (intern->logger)
     {
         Delete intern->logger;
         Intern->logger = NULL;
     }
     Zend_object_std_dtor(&intern->std);
}

Zend_object is a variable-length structure. So in the structure of the custom object, Zend_object needs to be placed in the last item:

(5) array

The hash table in 7.0 is defined as follows, with some comments:
/* 7.0 hash table structure */
Typedef struct _Bucket { /* An entry in the hash table */
Zval val; /* Delete element zval type is marked IS_UNDEF */
Zend_ulong h; /* hash value (or numeric index) */
Zend_string *key; /* string key or NULL for numerics */
Bucket;
Typedef struct _zend_array HashTable;
Struct _zend_array {
    Zend_refcounted_h gc;
    Union {
        Struct {
            ZEND_ENDIAN_LOHI_4(
                Zend_uchar flags,
                Zend_uchar nApplyCount,
                Zend_uchar nIteratorsCount,
                Zend_uchar reserve)
        } v;
        Uint32_t flags;
    } u;
    Uint32_t nTableMask;
    Bucket *arData; /* saves all array elements */
    Uint32_t nNumUsed; /* How much length is currently used, */
    Uint32_t nNumOfElements; /* The number of elements actually saved in the array. Once the value of nNumUsed reaches nTableSize, PHP will try to adjust the arData array to make it more compact. The specific way is to discard the entry of type UDENF */
    Uint32_t nTableSize; /* The size of the memory allocated to the array is a power of 2 (minimum of 8) */
    Uint32_t nInternalPointer;
    Zend_long nNextFreeElement;
    Dtor_func_t pDestructor;
};

Where PHP7 defines a series of macros in ZEND_HASH.H to manipulate arrays, including traversing key, traversing value, traversing key-value, and so on, here is a simple example:

/* Array example */
Zval *arr;
Zend_parse_parameters(ZEND_NUM_ARGS() , "a", &arr_qos_req);
If (arr)
{
     Zval *item;
     Zend_string *key;
     ZEND_HASH_FOREACH_STR_KEY_VAL(Z_ARRVAL_P(arr), key, item) {
         /* ... */
     }
}
/* After getting the item, you can get the long, double, and string values through the following apis.
Zval_get_long(item)
Zval_get_double(item)
Zval_get_string(item)

In the PHP5.6 version, you find the key by Zend_hash_find and then give the result to the zval * * variable, and you need to allocate memory yourself when the query is not available, initialize an item, and set the default value.

2. API changes in PHP7 (1) Duplicate parameters

PHP5.6 in many of the API needs to fill in a duplicate parameter, indicating whether a variable needs to be copied a copy, especially the operation of the string class, PHP7.0 cancel the duplicate parameter, for string related operations, as long as there are duplicate parameters, delete directly. Because the zval_string structure is defined in PHP7.0, the operation of the string no longer requires the duplicate value, and the underlying directly uses Zend_string_init to initialize a zend_string. In PHP5.6, strings are stored in Zval, and Zval's memory needs to be allocated manually.

The APIs involved are summarized below:

Reference

Add_index_string, Add_index_stringl, ADD_ASSOC_STRING_EX, ADD_ASSOC_STRINGL_EX, add_assoc_string, Add_assoc_stringl , Add_next_index_string, Add_next_index_stringl, ADD_GET_ASSOC_STRING_EX, ADD_GET_ASSOC_STRINGL_EX, ADD_GET_ASSOC_ String, Add_get_assoc_stringl, add_get_index_string, Add_get_index_stringl, ADD_PROPERTY_STRING_EX, add_property_ STRINGL_EX, Add_property_string, Add_property_stringl, zval_string, Zval_stringl, retval_string, RETVAL_STRINGL, Return_string, Return_stringl

(2) Make_std_zval

In PHP5.6, zval variables are allocated on the heap, creating a Zval variable requires declaring a pointer and then allocating space using Make_std_zval. PHP7.0, this macro has been canceled, the variable is allocated on the stack, directly define a variable, no longer need to make_std_zval, the use of the place, directly remove the good.

(3) Zend_rsrc_dtor_func

Modify parameter name rsrc to Res

/* PHP5.6 */
typedef struct _zend_rsrc_list_entry {
    void *ptr;
    int type;
    int refcount;
} zend_rsrc_list_entry;
typedef void (*rsrc_dtor_func_t)(zend_rsrc_list_entry *rsrc TSRMLS_DC); #define ZEND_RSRC_DTOR_FUNC(name)        void name(zend_rsrc_list_entry *rsrc TSRMLS_DC) /* PHP7.0 */
struct _zend_resource {
    zend_refcounted_h gc;/*7.0中对引用计数做了结构封装*/
    int               handle;
    int               type;
    void             *ptr;
};
typedef void (*rsrc_dtor_func_t)(zend_resource *res); #define ZEND_RSRC_DTOR_FUNC(name) void name(zend_resource *res)

PHP7.0, the zend_rsrc_list_entry structure is upgraded to Zend_resource, and only the parameter names can be modified in the new version.

(4) Level two pointer macro, i.e. Z_*_PP

PHP7.0 all the PP macros are canceled, most of the cases directly using the corresponding P-macro.

(5) Zend_object_store_get_object was canceled

According to the official wiki, you can define the following macros to get object, in fact, this macro is still more frequent:

static inline user_object *user_fetch_object(zend_object *obj) {
    return (user_object *)((char*)(obj) - XtOffsetOf(user_object, std));
}
/* }}} */ 
#define Z_USEROBJ_P(zv) user_fetch_object(Z_OBJ_P((zv)))

(6) Zend_hash_exists, Zend_hash_find

For all functions that require string arguments, the way in PHP5.6 is to pass two arguments (char* + len), and zend_string is defined in PHP7.0, so only one zend_string variable is required.

The return value becomes the Zend_bool type:

/* example */
Zend_string * key;
Key = zend_string_init("key",sizeof("key"), 0);
Zend_bool res_key = zend_hash_exists(itmeArr, key);  

Reference

Resources

1. php5 to Phpng:http://yaoguais.com/?s=md/php/php7-vm.md

2. php extension Development and kernel application: http://www.walu.cc/phpbook/10.1.md

3. New Hashtable implementation and performance improvements in PHP 7: http://gywbd.github.io/posts/2014/12/php7-new-hashtable-implementation.html

4. In-depth understanding of PHP7 's zval:https://github.com/laruence/php7-internal/blob/master/zval.md

5. Official wiki:https://wiki.php.net/phpng-upgrading

6. PHP Manual: http://php.net/manual/zh/index.php

7. PHP7 use of resources to package the implementation of third-party extensions and their source code interpretation: https://mengkang.net/684.html

Five, AMS platform upgrade PHP7 performance Optimization results

Current Network Service is a very important and sensitive environment, the impact of the user experience, the heavy production of the current network accident. As a result, we completed the PHP7 compilation and testing work in late April, in one of the AMS machines were gray-line, observed a few days later, and then gradually expand the gray scale, in early May to complete the upgrade.

This is what we're measuring. AMS a query for multiple activity counters of the test results, as well as the current network CGI machine, the peak of the same TGW traffic scenario CPU load data:

As far as our business is measured and the net results are, it is basically consistent with what the authorities say is a performance boost.

AMS platform has a lot of CGI machines, PHP7 upgrade and application to bring us a performance improvement, can effectively save the cost of hardware resources. And, through the Apache2.4 event mode, we've also enhanced Apache's ability to support concurrency.

Vi. Summary

We PHP7 upgrade research and development project team, in the past a relatively long period of time, through continuous efforts and progress, finally in April 2016 late in the net gray, early May in the cluster of full-scale upgrade, for our AMS activity operating platform to bring a significant performance improvement.

PHP7 's innovation, for the PHP language itself, has extraordinary significance and value, which makes me more convinced that PHP will be a better language. At the same time, thanks to the developers of the PHP community for the performance improvements we have brought to our business.

Daily request million level QQ member AMS platform PHP7 Upgrade Practice