Ceph source code analysis: Network Module

Ceph source code analysis: Network Module

Because Ceph has a long history, the network does not use the epoll model that is currently commonly used, but uses a multi-threaded model similar to MySQL. Each connection (socket) has a read thread, read data from the socket continuously. A write thread writes data to the socket. Multi-threaded implementation is simple, but the concurrency performance is not flattering.

Messenger is the core data structure of the network module and is responsible for receiving/sending messages. OSD mainly has two types of Messenger: ms_public messages with clients, and ms_cluster processes messages with other osdks.

Data Structure

The core of the network module is SimpleMessager:

(1) it contains an Accepter object, which creates a separate thread for receiving new connections (Pipe ).

Void * Accepter: entry (){... int sd =: accept (listen_sd, (sockaddr *) & addr. ss_addr (), & slen); if (sd> = 0) {errors = 0; ldout (msgr-> cct, 10) <"accepted incoming on sd" <sd <dendl; msgr-> add_accept_pipe (sd );... // create a new PipePipe * SimpleMessenger: add_accept_pipe (int sd) {lock. lock (); Pipe * p = new Pipe (this, Pipe: STATE_ACCEPTING, NULL); p-> sd = sd; p-> pipe_lock.Lock (); p-> start_reader (); p-> pipe_lock.Unlock (); pipes. insert (p); accepting_pipes.insert (p); lock. unlock (); return p ;}

(2) contains all connection objects (Pipe). Each connection Pipe has a read/write thread. The read thread reads data from the socket and then places the message in the DispatchQueue distribution queue. The write thread is responsible for extracting the Message from the sending queue and writing it to the socket.

Class Pipe: public RefCountedObject {/*** The Reader thread handles all reads off the socket -- not just * Messages, but also acks and other protocol bits (sort ting startup, * when the Writer does a couple of reads ). * All the work is implemented in Pipe itself, of course. */class Reader: public Thread {Pipe * pipe; public: Reader (Pipe * p): pipe (p) {} void * entry () {pipe-> reader (); return 0 ;}} reader_thread; // read thread friend class Reader;/*** The Writer thread handles all writes to the socket (after startup ). * All the work is implemented in Pipe itself, of course. */class Writer: public Thread {Pipe * pipe; public: Writer (Pipe * p): pipe (p) {} void * entry () {pipe-> writer (); return 0 ;}} writer_thread; // write thread friend class Writer ;... /// send queue map <int, list <Message *> out_q; // priority queue for outbound msgs DispatchQueue * in_q; // receive queue

(3) it contains a distribution queue DispatchQueue. The distribution queue has a special distribution thread (DispatchThread) that distributes messages to Dispatcher (OSD) for specific logic processing.

Message receipt

The receiving process is as follows: the read thread of Pipe reads the Message from the socket, puts it into the receiving queue, and then the delivery thread obtains the Message and sends it to the Dispatcher for processing.

Message sending

The sending process is as follows:

Other materials

This article parses Ceph: the processing of the network layer briefly introduces the Ceph network, but the relationship between Pipe and Connection does not seem accurate. Pipe is the encapsulation of socket, connection is more advanced and abstract.

Install the distributed storage system Ceph on CentOS 7.1

Ceph environment configuration document PDF

Deploying Ceph on CentOS 6.3

Ceph Installation Process

HOWTO Install Ceph On FC12 and FC Install Ceph Distributed File System

Ceph File System Installation

CentOS 6.2 64-bit installation of Ceph 0.47.2

Ubuntu 12.04 Distributed File System (Ceph)

Install Ceph 0.24 on Fedora 14

Ceph details: click here
Ceph: click here

This article permanently updates the link address: