Core network simulation software analysis

Core uses LXC (Linux namespace Container) technology and bridge technology to achieve virtual host and virtual network simulation. LXC uses the Process Group resource Management framework in the Cgroups subsystem to implement a virtual host as a relatively independent process for the same group. LXC has been added to the kernel 2.6.28 version, core to the virtual host creation and management, through the C language system calls to the implementation, the specific code implemented in the Core/daemon/src folder.

File list

File name	Function description
Vnoded_main.c	Creating a daemon that runs LXC container vnoded (PID 1)
Vcmd_main.c	Tool, specifying LXC container Run command
Netns_main.c	Tool, creating a LXC container to run the specified program
Netns.h, NETNS.C	Provides LXC container management operations
Netnsmodule.c	A Python library that provides LXC container management operations
Vnode_server.h,. C
Vnode_client.h,. C
Vnode_cmd.h,. C
Vnode_msg.h,. C
Vnode_io.h,. C	Provides basic IO operations
Vnode_chnl.h,. C	Provides connection management operations for the vnoded process

LXC container (netns.h, NETNS.C)

The LXC container is created under Linux through process cloning (syscall call). Each LXC container PID, IPC, network and other system resources are no longer global, but belong to a specific namespace, each namespace in the resources relative to other namespace is transparent.

PID = Syscall (Sys_clone, Flags | NSCLONEFLGS, NULL, NULL, NULL, or NULL)

The corresponding flag needs to be specified when creating a new LXC container.

#define NSCLONEFLGS \//NETNS.C

(                                      \

SIGCHLD | \

clone_newns | \

clone_newuts | \

CLONE_NEWIPC | \

Clone_newpid | \

Clone_newnet \

)

L SIGCHLD indicates that a SIGCHLD signal is sent to the parent process after the creation process exits

L Clone_newns represents a separate file hierarchy view for creating a process setting

L clone_newuts indicates that the creation process sets a separate host name

L CLONE_NEWIPC to create a process that sets up an independent IPC environment

L clone_newpid to create a process to set up a separate PID environment

L Clone_newnet represents the creation of a process-independent network environment

The above flags can be combined to create LXC containers as needed.

vnoded process (VNODED_MAIN.C)

The vnoded process is the first process that runs after the virtual host is created (pid=1), which is created by the Pycore call/usr/sbin/vnoded command, and the parameters that are required to be supplied when the command executes are newnetns, Ctrlchnlname, LogFileName , Pidfilename and so on. The vnoded process is a daemon that executes when it enters the message loop until it exits.

Ev_loop (vnodeserver->loop, 0);

The vnoded process is equivalent to the operating system running within the LXC container, so it provides simple operating system functions: File management, user access, process management three basic functions. vnoded Process execution is divided into three steps:

Step1. Create user Access CTRLCHNL .

Core virtual nodes need to be accessed through CTRLCHNL, CTRLCHNL is actually the sock_seqpacket type of socket interprocess communication mechanism. CTRLCHNL is established before a process clone, so that the cloned process (the child process) can naturally access the socket handle.

FD = socket (Af_unix, sock_seqpacket, 0)

Bind (FD, (struct sockaddr *) &addr, sizeof(addr))

Step2. Create a container ( namespace ).

PID = nsfork (0);

Child process:

Close all files opened by the parent process; Set input/output.

for (i = 3; i < OpenMAX; i++)

if (i! = CTRLFD) Close (i);

...

Dupfile ("/dev/null", O_rdonly, Stdin_fileno);

Setvbuf (stdout, NULL, _iolbf, 0);

Setvbuf (stderr, NULL, _iolbf, 0);

Parent process:

Exit wait.

_exit (0); /* Nothing else for the parent to do */

Step3. establish a listening message loop.

The vnoded process (running in a container) establishes two message loops, one is the CTRLCHNL message loop, and the user command is received. The other is a child process message that receives the child process information. The implementation of these two message loops is implemented using the Libev library, encapsulated in a vnode_server_t structure.

Server = Vnode_newserver (ev_default_loop (0), CTRLFD, ctrlchnlname);

Vcmd Command (VCMD_MAIN.C)

The Vcmd command is used to specify the container to run the program, specifically to send a message to the vnoded process, so that it clones a new process. The configuration of the NIC and the link needs to be done via the Vcmd command. In order to specify the LXC container, Vcmd needs to specify the ctrlchnlof the container when executing.

Linux programs require standard input and output and error outputs, either pipe or terminal (PTY), collectively referred to as I/O. To execute another program with the Vcmd command, you need to specify I/O for the program, through the parameters-I,-I,-Q, and so on. The VMD defines three input and output I/O types: file, pipe, terminal, V4.5 version only implements the terminal type, namely Vcmd_io_pty.

typedef enum {

Vcmd_io_none = 0,

VCMD_IO_FD,

Vcmd_io_pipe,

Vcmd_io_pty,

} vnode_client_cmdiotype_t;

The vcmd command execution is divided into the following four steps.

Step1. initializing input and output .

Initializes the terminal and pipeline according to the command parameters.

Vcmd.cmdio = Vnode_open_clientcmdio (Iotype);

Step2. Establish a container access client.

The container access client is encapsulated in the vnode_client_t structure. The Vcmd command creates the vnode_client_t structure and accesses it with the server process (vnoded).

Vcmd.client = vnode_client (ev_default_loop (0), Ctrlchnlname,

VCMD_IOERRORCB, &vcmd);

Step3. establishes a command reply response message loop.

The vcmd command process executes, receives the user input, passes the input to the process in the container, and also displays the process execution reply in the container at the user terminal, so Vcmd needs to establish two message loops: its own I/O message loop and the I/O message loop of the process in the container.

VCMD->STDIN_FWDFD = vcmd->cmdio->stdiopty.masterfd;

Vcmd->stdin_watcher.data = &vcmd->stdin_fwdfd;

Ev_io_init (&vcmd->stdin_watcher, VCMD_RWCB,

Stdin_fileno, Ev_read);

Ev_io_start (Loop, &vcmd->stdin_watcher);

VCMD->PTYMASTER_FWDFD = Stdout_fileno;

Vcmd->ptymaster_watcher.data = &vcmd->ptymaster_fwdfd;

Ev_io_init (&vcmd->ptymaster_watcher, VCMD_RWCB,

VCMD->CMDIO->STDIOPTY.MASTERFD, Ev_read);

Ev_io_start (Loop, &vcmd->ptymaster_watcher);

Step4. sends a command message.

Vcmd through ARGC, argv obtains the name and parameters of the program that the user wants to execute in the container, and then sends it to the vnoded process via CTRLCHNL , which is then executed by the vnoded process through forkexec. For programming convenience, vcmd as the following levels of abstraction.

Figure 1 Vcmd message passing mechanism

Command-level cmd to the user Client vnode_client, the user client to the message layer msg, and then through the socket sent to the Vnode process message layer msg, up to the service client, and then by the command-level cmd unpacking command parameters for execution. In addition, VCMD has established standard input and output with child processes.

vnode_server_t Structural Body (vnoded_server.h,. c)

The system service function initiated by the vnoded process is for the receiving user to request a connection, executes (forkexec) the corresponding program in the container according to the user command, and returns the program result to the user. Because the user may have more than one, the execution of the program will have more than one, the vnode_server_t structure is stored with the queue clientlist,cmdlist. The SERVERFD is used to store the socket established by the CONTRLCHNL of the user connection bindings. Fdwatcher and Childwatcher respectively establish two message observers, which observe the user connection message, and then observe the child process execution message.

typedef struct {

Tailq_head (Clientlist, cliententry) Clientlisthead;

Tailq_head (Cmdlist, cmdentry) Cmdlisthead;

struct ev_loop *loop;

Char Ctrlchnlname[path_max];

Char Pidfilename[path_max];

int serverfd;

Ev_io Fdwatcher;

Ev_child Childwatcher;

} vnode_server_t;

User connection message loop

The user connection message is handled by the callback function VNODE_SERVER_CB.

static void Vnode_server_cb (struct ev_loop *loop, ev_io *w, int revents)

The function generates a Vnode_client object, joins to the clientlist queue, and initiates the Msgio message loop of the Vnode_client object, after which the client object and the server's IO processing are handed over to the Msgio message loop processing.

Vnode_msgiostart (&client->msgio, Server->loop, CLIENT->CLIENTFD, client, Client_ioerror, Msghandler)

Child process message loop

The child process message is handled by the callback function VNODE_CHILD_CB.

static void Vnode_child_cb (struct ev_loop *loop, ev_child *w, int revents)

The function sends state information to the user based on the status returned by the child process.

Vnode_send_cmdstatus (CLIENT->CLIENTFD, Cmd->cmdid, W->rstatus)

After that, the corresponding CMD is removed from the cmdlist.

Tailq_remove (&server->cmdlisthead, cmd, entries);

vnode_client_t Structural Body (vnoded_client.h,. c)

The vnode_client_t struct is used to create a user client entity that corresponds to a vnode_cliententry_t table entry in the vnoded process. vnode_client_t receives the user command vcmd_t, encapsulates the user command in the Vnode_msgio structure, and Vnode_msgiostart initiates the message loop.

typedef struct vnode_client {

Tailq_head (Cmdlist, cmdentry) Cmdlisthead;

struct ev_loop *loop;

int serverfd;

struct Vnode_msgio Msgio;

void *data;

vnode_clientcb_t IOERRORCB;

int32_t Cmdid;

} vnode_client_t;

Cmdlist is used to store vcmd_t, but to understand that the previous command corresponds to a user client and does not need to be stored in a list, the check code finds that no object is inserted into the list, vcmd_t exists in data. The SERVERFD is used to store vnoded process CTRLCHNL. Msgio stores the MSG message layer structure body Vnode_msgio. The IOERRORCB is used to store server-side IO error callback functions. The Cmdid is used to store the child process ID number.

The user client sends a message to the service client and should specify a message response callback to the message layer. The program defines four types of messages.

typedef enum {

Vnode_msg_none = 0,

Vnode_msg_cmdreq,

Vnode_msg_cmdreqack,

Vnode_msg_cmdstatus,

Vnode_msg_cmdsignal,

Vnode_msg_max,

} vnode_msgtype_t;

However, only two types of callbacks are actually specified.

static const vnode_msghandler_t Msghandler[vnode_msg_max] = {

[Vnode_msg_cmdreqack] = Vnode_clientrecv_cmdreqack,

[Vnode_msg_cmdstatus] = Vnode_clientrecv_cmdstatus,

};

vnode_msgio_t Structural Body (vnoded_msg.h,. c)

Vnode_msgio_t is responsible for sending and receiving messages, it maintains the focus of FD processing, but is not responsible for the initialization of FD, and Msghandler is also specified by the client.

typedef struct Vnode_msgio {

struct ev_loop *loop;

int fd;

Ev_io Fdwatcher;

vnode_msgbuf_t Msgbuf;

void *data;

vnode_msghandler_t IOError;

vnode_msghandler_t Msghandler[vnode_msg_max];

} vnode_msgio_t;

The two most critical functions of vnode_msgio_t are send and receive message functions.

ssize_t vnode_sendmsg (int fd, vnode_msgbuf_t *msgbuf);

ssize_t vnode_recvmsg (vnode_msgio_t *msgio);

Vnode_sendmsg can be called externally, the message vnode_recvmsg by Fdwatcher in the VNODE_MSG_CB callback function when the message is triggered, after reading the message according to the message type, choose different msghandler processing.

Msghandlefn = msgio->msghandler[msgio->msgbuf.msg->hdr.type];

Msghandlefn (Msgio);

Vnode_cmd.h,. C, vcmdmodule.c

struct vnode_cmdentry_t is defined, but no reference is made (the visible code is modified on other code). Defines a command-level transceiver function. The main is to send command request Vnode_send_cmdreq, receive command request Vnode_recv_cmdreq, send command state vnode_send_cmdstatus, send command signal vnode_send_cmdsignal, Receive command signal vnode_recv_cmdsignal.

void vnode_recv_cmdreq (vnode_msgio_t *msgio);

int vnode_send_cmdreq (int fd, int32_t cmdid, char *argv[], int infd, int OUTFD, int errfd);

int vnode_send_cmdstatus (int fd, int32_t cmdid, int32_t status);

int vnode_send_cmdsignal (int fd, int32_t cmdid, int32_t Signum);

void vnode_recv_cmdsignal (vnode_msgio_t *msgio);

Some functions are defined only, and are not called, and are estimated in perfection, such as Vnode_send_cmdstatus and

Vnode_recv_cmdsignal.

Core network simulation software analysis