Qihoo 360 and Go

This is a creation in Article, where the information may have evolved or changed.

In China, Qihoo 360 is a major provider of internet and mobile security products and services, and by June 2014 Qihoo has 500 million PC active users and more than 640 million mobile users. Qihoo is also running China's most popular web browser and PC search engine (the original).

My team, the push service team, provides services (PC and mobile) for more than 50 company products, including thousands of applications placed on our open platform.

Our preference for go begins with the first attempt to provide push functionality for a product of Qihoo in 2012. The initial Nginx + LUA + Redis solution was not able to meet our real-time performance requirements because of the heavy load. In this case, the latest release of the 1.0.3 Go has caught our attention, and with the Goroutine and channel features it provides, we have developed a prototype within a few weeks. Our system was initially run on 20 servers, capable of processing 20 million live connections and sending 2 million of messages per day. The system now runs on more than 400 servers, supports 200 million live connections, and sends more than 10 billion messages per day.

With the rapid expansion of business and the growing demand for our push services, the initial go system quickly encountered a bottleneck: The heap size reached 69G and the GC paused up to 3-6 seconds. Even more troubling is that we have to restart the system every week to free up memory. To be honest, we've considered abandoning go and rewriting all of the core components in C to replace them. However, as we have not, we are having trouble migrating the code of the business logic layer. As a result, as the only person at the time, I was not able to ensure that the business logic migrated smoothly to the C service Framework while maintaining the go system.

So I decided to stay on the go system (probably my most witty choice) and soon made significant progress. Here are a few points and tips we summarize:

① replaces short connections with long connections (using connection pooling), reducing the creation of buffers and objects during communication.
② uses object pools and memory pools appropriately to reduce the load on the GC.

③ uses a task pool, in which a set of processes is used as a consumer to poll and perform a global task or a task from a message queue-a task that is sent as a producer by a co-owner that holds the connection, instead of generating a temporary association for each task.
④ Monitor and control the number of processes in the program. Lack of control can overwhelm the GC, for example, because the peak of the number of threads that occur with uncontrolled acceptance of external requests is applied to the GC, and RPC calls destined for internal services may block newly created threads.
⑤ in the mobile network, remember to set the read-write timeout limit to the connection, otherwise it may cause the association to block. Use this setting sparingly in a local area network, or it will reduce the efficiency of RPC communication.
⑥ uses pipeline (under the TCP full-duplex feature) to enhance the communication efficiency of the RPC framework.
Finally, we succeeded in iterating through the architecture three times, with two iterations of the RPC framework or limited staffing. Thanks to the convenience of Go development, here is our current system architecture:

The process of continuous optimization can be explained by the following table:

Also, after these optimizations, there is no need to temporarily release the memory or reboot the system.

Even more exciting, we've developed a visual platform that reflects the Go program's running profile in real time online. Now we can easily get the system status and make a diagnosis to eliminate the hidden dangers. This is the system in operation:

This platform can also do a great thing--simulating the connection and operation of millions of users online. By using this Distributed stress test tool (also implemented with Go), and observing all of these intuitive real-time data. We evaluate the effectiveness of each optimization and locate the system bottleneck and then troubleshoot the problem. So far, we have tried almost every means of system optimization. We still expect the GC team to bring in more good news and let us get out of the heavy development work. Come to think of it, maybe someday our experience will become useless because of the continued development of GO.

That's why I ended this sharing by expressing my gratitude for the opportunity to attend Gopher China. This is an event that we go to learn, to share, and a chance to show the popularity and prosperity of go in China. Within Qihoo There are many other teams that are starting to learn about go, or are already trying to use go.

In the foreseeable future, I believe that China will have more internet companies to join us, use go to recreate their systems, and the results of the Go development team will make more developers and companies benefit.

Translation by Open source China: Qihoo 360 and go

The original point of this view