Qihoo and Go

This is a creation in Article, where the information may have evolved or changed.

Transferred from: Http://blog.golang.org/qihoo

Qihoo and Go

This guest blog post is written by Yang Zhou, software Engineer at Qihoo 360.

Qihoo major provider of the Internet and mobile security products and services in China, and operates a major Android -based Mobile distribution platform. At the end of June, Qihoo had about million monthly active PC Internet users and over 640 million mobile users. Qihoo also operates one of China's most popular Internet browsers and PC search engines.

My team, the Push Service team, provides fundamental messaging services for more than products across the company (both PC and mobile), including thousands of Apps in our open platform.

Our "Love affair" with Go dates back to + when we first attempted to provide push services for one of Qihoo's products. The initial version is built with Nginx + LUA + Redis, which failed to satisfy we requirement for real-time performance Due to excessive load. Under these circumstances, the newly-published Go 1.0.3 release came to our attention. We completed a prototype in a matter of weeks, largely thanks to the goroutine and channel features it provided.

Initially, our go-based system ran on the servers, with the million real-time connections in total. The system sent 2 million messages a day. That system now runs on servers, supporting million+ real-time connections. It now sends over ten billion messages daily.

With rapid business expansion and increasing application needs for our push service, the initial Go system quickly reached Its bottleneck:heap size went up to 69G, with maximum garbage collection (GC) pauses of 3-6 seconds. Worse Still, we had to reboot the system every week to release memory. It wouldn ' t is honest if we didn ' t consider relinquishing Go and instead, re-writing the entire core component with C. How Ever, things didn ' t go exactly as we planned, we ran into trouble migrating the code of Business Logic Layer. As a result, it is impossible for the personnel at (myself) to maintain the GO system while ensuring the L Ogic transfer to the C service framework.

Therefore, I made the decision to stay with Go system (probably the wisest one I had to make), and great headway was made Soon enough.

Here is a few tweaks we made and key take-aways:

• Replace short connections with persistent ones (using a connection pool), to reduce creation of buffers and objects during Communication.
• Use Objects and Memory pools appropriately, to reduce the load on the GC.

• Use a task Pool, a mechanism with a group of long-lived goroutines consuming global tasks or message queues sent by connect Ion Goroutines, to replace short-lived Goroutines.

• Monitor and control Goroutine numbers in the program. The lack of control can cause unbearable burden on the GC, imposed by surges in goroutines due to uninhibited acceptance O F external requests, as RPC invocations sent to inner servers may block Goroutines recently created.

• Remember to add read and write deadlines to connections when under a mobile network; Otherwise, it may leads to goroutine blockage. Apply it properly and with caution when under a LAN network, otherwise your RPC communication efficiency would be hurt.

• Use Pipeline (under full Duplex feature of TCP) to enhance the communication efficiency of the RPC framework.

As a result, we successfully launched three iterations of our architecture, and both iterations of our RPC framework even W ITH limited human resources. This can all attributed to the development convenience of Go. Below you can find the up-to-date system architecture:

The continuous improvement journey can be illustrated by a table:

Also, no temporary release of memory or system reboot is required after these optimizations.

What's more exciting are we developed an on-line real-time Visibility Platform for profiling Go programs. We can now easily access and diagnose the system status, pinning down any potential risks. Here is a screens shot of the system in action:

The great thing about this platform is so we can actually simulate the connection and behavior of millions of online use RS, by applying the distributed Stress Test Tool (also built using Go), and observe all real-time visualized data. This allows us to evaluate the effectiveness of any optimization and preclude problems by identifying system bottlenecks.

Almost every possible system optimization have been practiced so far. And we look forward to more good news from the GC team so that we could is further relieved from heavy development work. I Guess our experience is also grow obsolete one day and as Go continues to evolve.

Want to conclude my sharing by extending my sincere appreciation to the opportunity to attendgopher China. It is a gala for us to learn, to share and for offering a window showcasing Go's popularity and prosperity in China. Many other teams within Qihoo has already either got to know go, or tried to use go.

I am convinced that many more Chinese Internet firms would join us in re-creating their system in Go and the Go team ' s EFFO RTS'll benefit more developers and enterprises on the foreseeable future.

by Yang Zhou