Secrets of Google's IT architecture strategy

Secrets of Google's IT architecture strategy

 

Google is different. Its uniqueness is not only manifested in innovative thinking and creative applications (such as the Earth model in the lobby), but also in its unconventional IT strategy ......

Mountain View, Calif. (Google, hereinafter referred to as Google) has a building 43 at its headquarters. The building's central screen displays a Google Earth) similar map of the world, a rotating Earth constantly flashing colorful points, like the thousands of candle lights of the Rome Court, every flash marks a new search launched by a Google user in this corner of the earth.

This also means that Google once again satisfies people's curiosity and desire for unknown information.

Google is different. Its uniqueness is not only manifested in innovative thinking and creative applications (such as the Earth model in the lobby), but also its unique IT strategy. From common sense, simple hardware products and free software cannot build an empire, but Google does. After performance adjustment, Google turned them into an incomparable distributed computing platform that supports large-scale search and emerging applications. We originally thought that these applications are personal consumption-level, but Google has changed everything. Now the business world is also using them, which makes the search company so distinctive.

The IT architecture behind the Google Web service may not be very important for countless search engine users, but it is Google's hundreds of users who are committed to organizing global information to achieve "ubiquitous access, the core work of target engineers is always available. This requires an IT blueprint that is fully consistent with Google's business vision in terms of coverage and ambition.

Google managers have been keeping silence on the company's IT strategy topics. They hate to talk about specific vendors or products. When asked about their servers and data centers, they always close their mouths. But after spending a day with several Google it leaders, we were able to reveal how the company's it works, it is not just a simple search engine running on numerous Server clusters. In its simple appearance, it implies a lot of internal R & D software, customized hardware, artificial intelligence, and persistent pursuit of performance and breaking the conventional human management model.

In terms of it philosophy, Google has a piece of advice for its peers: Try to avoid systems and software that everyone is using, and do things in their own way will have a unique competitive advantage.

"Enterprise culture determines your way of doing things ." Douglas "Douglas Merrill, vice president of Google engineering and de facto Chief Information Officer (CIO), pointed out," at this stage of our development, corporate concepts and culture are very different, this, in turn, inspires us to use a distinctive way to run systems that seem to be common to others."

Google's biggest it advantage is that it can create a high-performance system that is cost-effective (not cheap) and can withstand extremely high loads. Therefore, Stephen Arnold, an IT consultant, pointed out that Google and its competitors, such as Amazon, eBay, and Microsoft) compared with Yahoo (Yahoo) and other companies, it has a greater cost advantage. Google programmers are 50% more efficient than other web companies ~ 100%, because Google has developed a complete set of custom software libraries dedicated to supporting large-scale parallel system programming. He estimates that it may take four times for other competing companies to achieve the same effect.

Server Creation

How does Google achieve this? One of the measures, Merrill Lynch believes, is "because we build hardware by ourselves ." Google does not make computer systems, but it customizes hardware based on its own parameters, and then installs and adjusts the hardware system itself like Pimp My ride on MTV. Chris DiBona, open-source program manager, commented: "We are very good at buying commercial servers, transforming them for our use, and finally pushing performance to the extreme, so sometimes they get as hot as they want to melt."

This kind of self-built method comes from the frugal style inherent in the birth of Google from the garage. It is also closely related to the size of Google's super-large system. Good habits have continued so far. Google has 0.2 million to data centers ~ 0.45 million servers-this number may vary (depending on how you define the server and who will make the statistics ). However, what remains unchanged is the rising trend.

Google will not discuss these assets because it regards confidentiality as a competitive advantage. In fact, Google prefers open-source software because of its privacy. "If we purchase a software license or code license, people can guess Google's IT infrastructure as long as they sit on the right side ." "Using open-source software gives us an extra way to grasp our own destiny," DiBona said ."

Google prefers a large-scale server running mode. When there are hundreds of thousands of machines, the advantages of custom servers will multiply, and the effect will become more obvious. Google is building a 30-acre data center in Dales, a riverside of Columbia, Oregon, where it provides low-cost hydraulic power for computing and cooling (see Google data center self-contained).

Google organizes these servers running Linux operating systems in the form of cell, diBona compares this form to the "disk drive" of Internet services (but don't confuse it with the rumored Google storage service gdrive, "There is no such thing as gdrive." A Google spokesman made it clear .), The company's software programs are deployed in these non-expensive computer chassis, and the programmer determines their redundancy workload. This file system, consisting of many units, replaces commercial storage devices. DiBona says Google's unit devices are easier to build and maintain and suggest they can process larger volumes of data.

Google will not miss any attention to technical details. Over the years, our engineers have been studying the internal working mechanism of the microprocessor. As Google continues to grow, it will inevitably use specially customized and adjusted chips. Luiz Barroso, a well-known engineer, confirmed in a paper published in the industrial magazine last year that the major load of Google in recent years has been borne by a single-core system. However, many server-side applications, such as the Google search index service, require parallel computing to run poorly at the command level of a Single-core chip.

Barrosso, a chip designer at Digital Equipment and Compaq, believes that with AMD, Intel, and sun) starting to manufacture multi-core chips, more and more parallel computing at the chip level will emerge.

Google has also considered making its own computer chips, but from the industry trend, this risky action does not seem necessary. "The design of the microprocessor is complex and expensive," said Urs Holzle, senior vice president of operations. Google prefers to work with chip manufacturers to understand their applications and design suitable chips. This is a customer-suggested design, focusing on the overall throughput, performance, and power consumption ratio, rather than the peak performance of a single thread. "This is also the recent Design Trend and future direction of multi-core CPUs," holtzler said ."

Tailor-made software

To maximize hardware performance, Google has developed a considerable number of customized software. Innovative products mainly include mapreduce, a programming model for simplifying processing and creating large-scale datasets; bigtable, a system for storing and managing large-scale data; Sawzall, an interpreted programming language for analyzing large-scale datasets in a distributed computing environment; google File System for distributed file systems of data-intensive applications ); google workqueue is also used to process distributed system queue groups and task scheduling ).

It is precisely from these sawzall tools that reflect Google's persistent attention to computing efficiency. Not every company can solve efficiency problems from the bottom layer, but for Google, it is completely reasonable to design a programming language specifically for large-scale datasets that conventional relational databases cannot accommodate. Even if other programming tools can solve the problem, Google engineers will still develop a customized solution for efficiency. Google engineers believe that sawzall is comparable to mapreduce in C ++ and is easier to write.

Google's attention to efficiency makes it impossible to be satisfied with the standard Linux kernel. Google will run the modified kernel version as needed. By adjusting the underlying performance of Linux, Google engineers have solved a series of thorny problems, such as data corruption and data bottlenecks, on the basis of improving the overall system reliability. Modifications to the kernel also enable Google's computer cluster system to run faster with higher communication efficiency.

Of course, Google occasionally suffers system faults. Once a situation occurs, countless users will be affected. A system failure lasting 30 minutes three years ago affected 20% of search traffic.

Google has developed its own website server, but does not use open-source Apache servers, even though it has a market share of more than 60%. DiBona believes that Google's website servers can run on more hosts. For applications with large content and mutual dependency on Google sites, server Load balancer is far more powerful than Apache. At the same time, Google servers are more difficult to program than Apache in accessing dynamic web pages of databases using standard public gateway interfaces (CGI), but eventually run faster. "If we can squeeze 10% ~ With 20% performance, we can save more system resources, power and manpower ." DiBona points out in summary.

Google also designed its own customer relationship management (CRM) system to support its own bidding and click-based Internet advertising charging business. However, Google's attitude towards designing its own tools is not static. For example, in the accounting software, it uses Oracle's financial als software.

Merrill Lynch uses a fork as an example to illustrate that ready-made products can also bring value. However, in some cases, ready-made software products are not necessarily applicable. "Our culture has a profound impact on our operations at all levels," he said. "So we don't want the tools we buy to change our work style and culture ."