MooseFS source code analysis (2)-mfsmount Module

1. fuse Introduction

Fuse must be installed on the client. Mfsmount uses the fuse module to have the same operation portal as file operations.

Fuse provides an interface for file operations. fuse then listens to file operations on this interface, and pass these operations on the file to our own code for processing. That is to say, fuse provides a mount point and then listens to it. Once an operation (new, read, or modify) occurs, it gives control to the code, the code for these operations is implemented in the mfsmount module.

The fuse_lowlevel_ops structure in fuse is used to specify custom file operations. The structure defined in mfsmount is as follows:

1. mfsmount Module main. c code snippet

1. Static struct fuse_lowlevel_ops mfs_servers = {
2. . Statfs = mfs_statfs,
3. . Lookup = mfs_lookup,
4. . Getattr = mfs_getattr,
5. . Setattr = mfs_setattr,
6. . Mknod = mfs_mknod,
7. . Unlink = mfs_unlink,
8. . Mkdir = mfs_mkdir,
9. . Rmdir = mfs_rmdir,
10. . Symlink = mfs_symlink,
11. . Readlink = mfs_readlink,
12. . Rename = mfs_rename,
13. . Link = mfs_link,
14. . Opendir = mfs_opendir,
15. . Readdir = mfs_readdir,
16. . Releasedir = mfs_releasedir,
17. . Create = mfs_create,
18. . Open = mfs_open,
19. . Release = mfs_release,
20. . Flush = mfs_flush,
21. . Fsync = mfs_fsync,
22. . Read = mfs_read,
23. . Write = mfs_write,
24. . Access = mfs_access,
25. };

Mfs_statfs, mfs_lookup... Mfs_write and mfs_access are functions implemented by mfsmount. There are 23 file operations in total, and mfsmount must be implemented by yourself. When you perform file operations on the mount point registered by fuse (corresponding to/mnt/mfs by default), fuse uses the mfsmount implementation to perform these operations. For example, if we create a text.txt file in/mnt/mfs, fuse will call mfs_create to perform the operation (of course, the file name and other necessary parameters must be passed ), fuse does not care where our code is used to create specific files.

Let's take a look at the definitions of these file operation functions in the header file in mfsmount:

Mfs_fuse.h code snippet

1. Void mfs_access (fuse_req_t req, fuse_ino_t ino, int mask );
2. Void mfs_lookup (fuse_req_t req, fuse_ino_t parent, const char * name );
3. Void mfs_getattr (fuse_req_t req, fuse_ino_t ino, struct fuse_file_info * fi );
4. Void mfs_setattr (fuse_req_t req, fuse_ino_t ino, struct stat * stbuf, int to_set, struct fuse_file_info * fi );
5. Void mfs_mknod (fuse_req_t req, fuse_ino_t parent, const char * name, mode_t mode, dev_t rdev );
6. Void mfs_unlink (fuse_req_t req, fuse_ino_t parent, const char * name );
7. Void mfs_mkdir (fuse_req_t req, fuse_ino_t parent, const char * name, mode_t mode );
8. Void mfs_rmdir (fuse_req_t req, fuse_ino_t parent, const char * name );
9. Void mfs_symlink (fuse_req_t req, const char * path, fuse_ino_t parent, const char * name );
10. Void mfs_readlink (fuse_req_t req, fuse_ino_t ino );
11. Void mfs_rename (fuse_req_t req, fuse_ino_t parent, const char * name, fuse_ino_t newparent, const char * newname );
12. Void mfs_link (fuse_req_t req, fuse_ino_t ino, fuse_ino_t newparent, const char * newname );
13. Void mfs_opendir (fuse_req_t req, fuse_ino_t ino, struct fuse_file_info * fi );

Note that each function has a fuse_ino_t type parameter ino. Each file corresponds to a unique ino value. This value is required for fuse. In the directory structure of the file (in the masterserver metadata), this ino is used as the identifier of the file and folder. In the process of creating files and folders, the directory structure data is stored on masterserver, and the client does not have the data. The ino of the root directory (/mnt/mfs) is set to 0.

Take deleting a directory as an example to explain the code. The following function is the function corresponding to the "delete directory" Operation registered in the mfsmount module:

2. Delete the corresponding function in the directory

1. Void mfs_rmdir (fuse_req_t req, fuse_ino_t parent, const char * name ){
2. Uint32_t nleng;
3. Int status;
4. Const struct fuse_ctx * ctx;
5.  
6. /*...........*/
7.  
8. Ctx = fuse_req_ctx (req );
9. Status = fs_rmdir (parent, nleng, (const uint8_t *) name, ctx-> uid, ctx-> gid );
10. Status = mfs_errorconv (status );
11. If (status! = 0 ){
12. Fuse_reply_err (req, status );
13. } Else {
14. Fuse_reply_err (req, 0 );
15. }
16. }

The parameters of the mfs_rmdir function are transmitted by fuse. Note that the fs_rmdir of Line 1 will communicate with the masterserver to perform the real deletion operation.

In short, fuse is used to provide an intermediate layer to capture the file operations on the user (or program) on the mount point and forward them to mfsmount's own code for processing, then, the processing structure is fed back to the user. For users, reading and writing files on this mount point is the same as reading and writing files in normal folders, and users do not feel any difference.

2. Working Process of mfsmount

▲After mfsmount is started, the command line parameters and configurations are read first.

▲Use fuse to define the mount point and correspond to the file operation function implemented by mfsmount.

▲Set the connection parameters to masterserver.

▲Start thread fs_nop_thread is used to maintain the connection status with masterserver. Fs_receive_thread connects to the masterserver for communication, registers on the masterserver, and performs initialization.

▲Start to respond to the input from the client (transmitted by fuse ).

The following describes the process of responding to user input. input is the operations we perform to create and modify files. These operations are handed over to specific functions defined in fuse_lowlevel_ops, communicate with masterserver in these specific functions.

Each time you perform a file operation, a threc type data is generated:

Threc Structure

1. Typedef struct _ threc {
2. Pthread_t thid;
3. Pthread_mutex_t mutex;
4. Pthread_cond_t cond;
5. Uint8_t * buff;
6. Uint32_t buffsize;
7. Uint8_t sent;
8. Uint8_t status;
9. Uint8_t release; // cond variable
10. Uint32_t size;
11. Uint32_t cmd;
12. Uint32_t packetid;
13. Struct _ threc * next;
14. } Threc;

There are mainly some status fields, the most important of which is buff. The data in the buff will be sent to masterserver:

The first 12 bytes of a buff are three integers. The following space is used to store the specific sending information. The buff content is as follows:

▲ 1st ~ 4 bytes (store an integer) is a specific command that describes what data needs to be taken.

▲ 5th ~ 8 bytes (an integer), the size of the data sent.

▲ 9th ~ 12 bytes (containing an integer) indicate packetid, which is unique and corresponds to each operation.

▲ 13th ~ ** Bytes, with a length of 5th ~ 8 bytes.

Let's take a look at the code snippet when creating a directory (fs_makedir:

Fs_makedir code snippet

1. ...
2. Ptr = fs_createpacket (rec, CUTOMA_FUSE_MKDIR, 15 + nleng );
3. PUT32BIT (parent, ptr );
4. PUT8BIT (nleng, ptr );
5. Memcpy (ptr, name, nleng );
6. Ptr + = nleng;
7. PUT16BIT (mode, ptr );
8. PUT32BIT (uid, ptr );
9. PUT32BIT (gid, ptr );
10. Ptr = fs_sendandreceive (rec, MATOCU_FUSE_MKDIR, & I );
11. ...

In the above 2nd lines of code, fs_createpacket will create a threc structure and set its buffer size to 15 + nleng + 12 bytes.

The following 12 bytes are fixed as the buffer header, which is filled in when the threc structure is created, as described earlier in the threc structure.

The content of the 15 + nleng bytes is the content sent to masterserver from the following lines 3rd, 4, 5, 7, 8, and 9. For different operations, the content sent to masterserver is different. The data returned by masterserver is also stored in the threc structure. The following is the content of the buffer in the threc structure corresponding to the Directory Creation operation when sending data to the masterserver:

After the data is returned from the masterserver, the data is retrieved from the threc structure and the folder parameters are handed over to fuse. The rest is that fuse uses these parameters to do its own thing (for example, display information to the user ).

3. Processing of reading and writing file data

Previously, when introducing fuse, I mentioned that mfsmount will define its own file operations. The read and write operations are special in dealing with actual data. Except read and write, all other operations only need to obtain data from masterserver. The read and write operations must first communicate with the masterserver to obtain the address and port of the chunkserver where the data resides, and then read and write data from the chunkserver. The threc structure is still used for communication between read and write and masterserver.

Obtain the IP address and Port of the chunkserver from the masterserver. This process is similar to other non-read and write operations. The main code is as follows:

Fs_readchunk code snippet

1. Ptr = fs_createpacket (rec, CUTOMA_FUSE_READ_CHUNK, 8 );
2. PUT32BIT (inode, ptr );
3. PUT32BIT (indx, ptr );
4. Ptr = fs_sendandreceive (rec, MATOCU_FUSE_READ_CHUNK, & I );
5. //......
6. GET64BIT (t64, ptr );
7. * Length = t64;
8. GET64BIT (t64, ptr );
9. * Chunkid = t64;
10. GET32BIT (t32, ptr );
11. * Version = t32;
12. If (I = 20 ){
13. * Csip = 0;
14. * Csport = 0;
15. } Else {
16. GET32BIT (t32, ptr );
17. * Csip = t32;
18. GET16BIT (t16, ptr );
19. * Csport = t16;
20. }

Masterserver returns the file version, length, ID, and chunkserver information. Mfsmount then communicates with the chunkserver to obtain data. After the data is read, the control is returned to fuse.