Pregel of graph database

/* Copyright notice: Can be reproduced arbitrarily, please be sure to indicate the original source of the article and the author information . */
Author: Zhang Junlin

Excerpt from "Big Data Day know: Architecture and Algorithms" Chapter 14, book catalogue here

Pregel is a large-scale distributed graph computing platform proposed by Google, which is designed to solve the problem of large scale distributed graph calculation in practical applications such as Web link analysis and social data mining.

1. Computational models

Pregel follows the BSP model on the conceptual model, and the entire calculation process consists of a number of super step steps that are executed sequentially, from a "super step" to the next "super step" until the algorithm's termination condition is reached (see Figure 14-13).

Pregel follows the graph node-centric pattern on the programming model, in Super step s , each graph node can summarize messages passed from other nodes in Super step s-1, change the state of the graph node itself, and send messages to other nodes, which are synchronized , it is received and processed by the other nodes in Super step S+1. The user only needs to customize a calculation function F (vertex)for the graph node, which can be used to compute the graph node, and other tasks, such as task assignment, task management, system fault tolerance, etc. are all referred to the Pregel system for implementation.

The typical Pregel calculation consists of the graph information input, the graph initialization operation, and the continuous execution super step separated by the global synchronization point, and the result can be output.

Each node has two states: active and inactive, at the beginning of the calculation, each node is active, with the calculation, some nodes to complete the calculation task to inactive state, if the node in the inactive state receives a new message, then again to active, If all the nodes in the diagram are inactive, the calculation task is completed and the Pregel output is calculated.

The following is a specific calculation task as an example of the Pregel graph calculation model, this task requires the maximum value of the node in the graph to propagate to all the other nodes in the graph, figure 14-14 is its, the graph of the real line arrow indicates the link relationship, and the figure in the node value represents the current value of the node, The dashed line in the diagram represents the message passing relationship between the different super-steps, and the graph node with the twill tag is an inactive node.

As can be seen, the value 6 is the maximum value in the figure, in the No. 0 step of the super step, all the nodes are active, the system executes the user function F(vertex): The node will be its own value through the link relationship, the node receives the message to select the maximum value, And compared with its own values, if larger than its own value, then update to a new value, if not larger than its own value, then to inactive state.

In the No. 0 Super step, each node propagates its own value through the link, the system enters the 1th super step, executes the F(vertex) function, and the first and fourth rows of nodes are updated to the new value 6 because they receive a value larger than their own. Nodes in the second and third rows do not receive a number larger than their own, so they become inactive. After executing the function, the node in the active state sends out the message again, the system enters the 2nd Super step, the second row node is inactive, because the new message is received, so the value is updated to 6, re-active, and the other nodes are in an inactive state. Pregel into the 3rd Super step, all the nodes are inactive, so the calculation task is finished, so the entire task is completed, the maximum value is passed through 4 super-steps to all the other nodes in the graph. Algorithm 14.1 is the Pregel C + + code that embodies this process.

2. System architecture

Pregel uses the "master-slave structure" to achieve the overall function, figure 14-15 is its architecture diagram, where a server acts as a "master server", responsible for the entire graph structure of the task segmentation, using "tangent edge" method to cut it into sub-graph (Hash (id) =ID MoD n ,n is the number of working servers, and assign tasks to a number of "Worker Servers", "Master server" command "work Server" to perform each super-step calculation, and to synchronize the points and collect the results of the calculation. The master server only manages the system and is not responsible for the specific graph calculations.

Each "worker server" is responsible for maintaining the state information assigned to its own sub-graph nodes and edges, and at the initial stage of the operation, all the graph node states are active, and the user-defined function F(Vertex) is called in turn for the node that is currently active. It is necessary to note that all data is loaded into memory for calculation. In addition, the work server manages the communication between the sub-diagrams maintained by this machine diagram and other work servers.

During subsequent computations, the "master server" notifies the "worker" through a command to start a super-step operation, and the "Worker server" calls F(Vertex) on the active node in turn, and when all active nodes are completed, the "Worker Server" notification master server "The number of active nodes remaining after this round calculation ends until all the graph nodes are inactive.

Pregel uses "Checkpoint" (CheckPoint) as its fault tolerance mechanism. Before the super step begins, the master server can command the worker to write its own data shard content to the storage point, including the node value, the edge value, and the message corresponding to the node.

The "master server" monitors the status of the "Worker server" through heartbeat monitoring, and when a "worker" fails, the "master server" assigns the corresponding data shard it is responsible for to another "worker", and the "work server" receiving the recalculation task reads the nearest "data shard from the storage point" Checkpoint to resume work, the checkpoint's super step may be several steps slower than the current system's super-step, at which point all the worker servers fall back to the super step that is consistent with the checkpoint to restart the calculation.

As can be seen from the above description, Pregel is a message-driven, synchronous graph computing framework that follows the graph node-centric programming model. Given the uniqueness and physical uniqueness of the "master server", it is clear that Pregel has the possibility of a single point of failure.

Think: In terms of fault-tolerant cycle selection, each super step can be done once, or you can choose a few super-steps, and then what are the pros and cons of each of these two approaches?

Answer: If you choose a short period of fault-tolerant measures, in the process of completing the task, the additional overhead will be more, but the advantage is that if the machine fails, the overall system fallback history is relatively close to the completion of the task as soon as possible, long-term fault-tolerant measures are the opposite, because the frequency is low, so the usual overhead However, if the machine fails, you need to back more super-step, leading to the execution of the elongated task. So there's a general tradeoff here too.

3. Pregel applications

This section describes how to construct a specific application under the Pregel framework through a number of common graph calculation applications.

(1) PageRank calculation

PageRank is an important reference factor in search engine sequencing, and its basic ideas and computational principles are explained earlier in this chapter and are not described here. The following is a C + + sample code that uses Pregel for PageRank calculations.

The Compute () function is the calculation function F(Vertex) that was described earlier for the node in the S Super step, and the user Vertex and rewrites Compute by inheriting the interface class ( messageiterator* msgs) interface function, you can quickly complete the application development, where messageiterator* msgs is the S-1 super-step to the current node of the message queue. The calculation function first accumulates the part of the PageRank score passed to the current node in the message queue, then obtains the current PageRank score of the graph node according to the calculation formula, if the current super step does not reach the loop termination condition 30 times, continues to pass the new PageRank value through the edge to the adjacency node, Otherwise, an end notification is issued, making the current node inactive.

(2) Single Source Shortest path

Finding the shortest path between nodes in the graph is a very common graph algorithm. The so-called "single-source Shortest path" means the shortest distance from any other node in the graph to the node, given the initial node StarTV. Below is an example of C + + code that calculates the single-source shortest path of a graph node under the Pregel platform.

As can be seen from the code, a graph node v from the previous super step received in the message queue to find the shortest path currently seen, if this value is smaller than the shortest path currently obtained by Node v , indicating that a shorter path is found, the update node value is the new shortest path, The new value is then propagated through the adjacency node, otherwise the current node is converted to an inactive state. After the calculation is complete, if the shortest path of a node is still marked as INF, it indicates that there is no accessible path between the node and the source node.

(3) Two-part chart maximum matching

Two The maximum matching is also a classic graph calculation problem, the following gives Pregel to use the random matching idea to solve the problem of a train of thought.

The above Pregel program uses random matching to solve the two-part maximum matching problem, each diagram node maintains a two-tuple: (' L/R ', matching node ID), ' L/R ' indicates whether the node is a left-end node in the two-part diagram or a right-side node, as an identification token. Another dimension of a binary group records the node ID on the match.

The algorithm runs through the following four stages.

Phase one: For nodes in the left side of the two graph that have not yet been matched, send a message to their neighboring nodes asking for a match and then into a non-active state.

Phase two: For nodes in the two graph that have not yet matched the right end, randomly select a receive from the received request matching message, and send a confirmation message to the left-side node receiving the request, then actively into the inactive state.

Phase three: After the left unmatched node receives the acknowledgment, it selects a node to receive, writes the matching node ID to indicate that it has been matched, and then sends a message to the right-side node that receives the request. Nodes that have already matched the left node do not have any action at this stage, because such nodes do not send any messages at all during the first phase.

Phase four: The right side has not yet matched the node to select at most one phase three sent over the request, and then writes the matching node ID to indicate that it has been matched.

With the continuous iteration of the four phases similar to the two-time handshake, a maximum matching result of two graphs can be obtained.

Pregel of graph database