Free dinner-Google technology learning

Author: Jiangnan Baiyi, source: Yun.

If Google's search engine is a free breakfast and Gmail is a free lunch,

Http://labs.google.com/papers/ is a free dinner from Google for developers.

However, looking at a table may not know where to start. It is not a pleasure to chew on English in a field that you are not familiar.

I. One ppt and four Chinese Translations

Fortunately, there was a guy interview with Google, who broke out and made a Google interal PPT:

Bytes.

Then there are nine times (http://sharp838.mblogger.cn) and the beauty of his father (http://my.donews.com/eraera/) on the road, the most important of the four papers translated:

• Mapredue: simple data processing on ultra-large clusters -- simplified data processing on large clusters
• The Google File System
• Massive Data Analysis: sawzall parallel processing -- interpreting the data: parallel analysis with sawzall
• Bigtable: Distributed Storage System of structured data -- a distributed storage system for structured data

2. Technical cornerstone of Google empire

Google empire is built on approximately 0.45 million servers, most of which are "cheap x86 boxes ". The 0.45 million servers are built on the following key infrastructure:

1. GFS (Google file system ):

GFS is a distributed file system suitable for large-scale distributed data processing applications. It is the foundation of Google's everything. Based on common hardware devices, gfs achieves Fault Tolerance Design and extremely high performance.

Kai-fu Lee said: Google's most powerful technology is its storage. I think computer science students should read this article (thanks again to the translation staff ).

It uses 64 m as a chunk (Block). Each chunk must exist on at least three machines. For the simple interaction process, see:

2. mapreduce

Mapreduce is a distributed programming mode for processing massive data sets, allowing programs to be automatically distributed to a super-large cluster composed of common machines for concurrent execution. Such as grep-style job and log analysis can be considered.

The run-time system of mapreduce solves the distribution details of input data, runs scheduling across machine clusters, processes machine failures, and manages communication requests between machines. This mode allows programmers to process resources of ultra-large distributed systems without having any experience in concurrent processing or distributed systems.

My own access to mapreduce is the path of Lucene-> nutch-> hadoop.
Hadoop is another masterpiece of Doug cutting, father of Lucene. It is a Java Distributed File System and MAP/reduce implementation.
The hadoop documentation is not detailed. Let's review the Google Chinese version of the paper. Everything is much clearer (thanks again to the translation team ).

Meng yan also has a clear blog: Map reduce-the free lunch is not over?

3. bigtable

Bigtable is a Google style database that uses structured files to store data.
Although it does not support relational data query, it is a distributed storage of large-scale structured data solution based on GFS/mapreduce.

Bigtable is a sparse, multidimensional, sorted map. Each cell consists of the row keyword, column keyword, and timestamp. The content of the cell is an uninterpreted string.
For example, the following table stores the reverse connection text between the content of each website and other websites.
Reverse url com. CNN. WWW (www.cnn.com) is the line keyword. The contents column stores the webpage content, and each content has a timestamp. Because there are two reverse connections, the archor column family has two columns: Anchor: cnnsi.com and anchhor: My. look. CA, the concept of the column family, so that the table can be horizontally expanded, the number of archor columns is not fixed.

For the sake of concurrent read/write, hot zone, and HA, bigtable certainly does not exist in comma-separated text files. It is stored in a database structure called sstable, there are two compression algorithms with different focuses: bmdiff and zippy.

4. sawzall

Sawzall is a domain language based on mapreduce and can be considered as a distributed awk. Its program control structure (if, while) is similar to the C language, however, its domain language semantics simplifies code that completes the same function by more than 10 times than mapreduce's c ++ code.

1 proto "cvsstat. proto"
2 submits: Table sum [hour: int] of Count: int;
3 log: changelistlog = input;
4 hour: Int = hourof (log. Time)
5 emit submits [hour] <-1;

Tianshu? Take a look.

This time, we will count the number of CVS submissions within 24 hours every day.
First, its variable definition is similar to Pascal (I: Int = 0; that is, it defines variable I, the type is int, and the initial value is 0)

1: Introduce the cvsstat. proto protocol description. For more information, see.
2: Define the int array submits to store the statistical results, and use hour as the subscript.
3. cyclically convert the file input to the changelistlog type and store it in the log variable. The type and conversion method are described in cvsstat. Proto.
4. Retrieve the hour value of the commit time log. Time in changlog.
5. emit aggregation: In the sumits result array, add 1 to the number of submissions of the hour, and then automatically cycle the next input.

Steps 1 and 2 are preparation and definition, Steps 3 and 4 are map, and step 5th is reduce.

Iii. Summary:

This article is just a brief introduction to Google's technical overview. We all know that in the future, we will not be able to talk about anything, but what we really want to learn is how to solve high concurrency in this paper, design ideas and details of high reliability .....