MFS theory Knowledge

Moosefs theory

Advantages of MFS:

1. Free (GPL)

2. Universal File System , No need to modify the upper application can be used

3. can be expanded online , the architecture is highly scalable

4. Simple Deployment

5. High Availability , can set any degree of file redundancy ( provides a higher level of redundancy than raid1+0 , and will never affect the performance of read or write , only speed !)

6. "Recycle Bin "Provides a system-level service Don't be afraid of the wrong operation. provides similar oralce flash back, etc. advanced dbms instant rollback feature !)

7. provide snapshot features for commercial storage such as NETAPP,EMC,IBM . ( A snapshot of the file can be created for the entire file, even in the file being written. )

8. a C implementation of Google filesystem

9. provide web GUI monitoring interface

increase the efficiency of random reads or writes

improve the reading and writing efficiency of large amount of small files .

Moosefs is a distributed file system

the Moosefs file system structure consists of the following four roles:

1. Management Server Managing Server (Master)

2. meta-data log server Metalogger server (Metalogger)

3. datastore Server data servers (chunkservers)

4. client mount using client computers

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/86/F9/wKioL1fQEPXwJEgtAAJ1F7MQ8fw828.png "title=" Picture 1.png "alt=" Wkiol1fqepxwjegtaaj1f7mq8fw828.png "/>

various role roles:

1. : Responsible for the management of each data storage server File read/write schedule file space reclamation and recovery multi-node copy.

2. : responsible for backup master server change log file changelog_ml.*.mfs, master server When something goes wrong, take the job.

3. : responsible for connecting to the Management Server obey the Management Server schedule Span style= "font-family: ' Times New Roman ';" >, provides storage space Span style= "font-family: ' The song Body '; > and provide data transfer to customers.

4. client : hook up the data storage server managed on the remote Management Server via the fuse Core interface , seemingly shared file system and local the Unix file system uses the same effect.

MFS Read-write principle :

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M02/86/FB/wKiom1fQES_DrzGTAACsQG0SvDg060.png "title=" Picture 2.png "alt=" Wkiom1fqes_drzgtaacsqg0svdg060.png "/>

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/86/F9/wKioL1fQEUuhDtB8AAEMWXhuNUc695.png "title=" Picture 3.png "alt=" Wkiol1fqeuuhdtb8aaemwxhunuc695.png "/>

The original read / write speed is obviously largely dependent on the performance of the hard disk being used , the capacity of the network, and the topology , the better the throughput of the hard disk and network used , The performance of the whole system will be better.

. Mfsmaster.lock : file Records the main process of a running Mfsmaster

Metadata.mfs, metadata.mfs.back moosefs file system metadata metadata mirroring

Changelog.*.mfs : is the moosefs file system metadata change log ( merged into Metadata.mfs once every one hours )

The size of the Metadata file is dependent on the number of files ( not their size ). the size of the changelog log depends on the number of operations per hour , But the length of time ( By default, by the hour ) is configurable.

to modify the limitations of the maximum file descriptor under Linux :

when a large number of small files are written , a serious error may occur that may be related to the operating system file descriptor. The operating system default file descriptor is 1024x768. 1.6.26 version defaults to 100000

when recommended to go live, master and chunker Modify the file descriptor system-level limit : It is the sum of the file descriptors that limit all users to open , You can change the limit by modifying the kernel parameters :

# vi/etc/sysctl.conf Add

fs.file-max=102400

If this value defaults to large enough, you do not have to change

# sysctl-p

command to make it effective.

user-level throttling : simply modify the user-level maximum file descriptor limit , which means that the total number of file descriptors that are executed after each user logon cannot exceed this limit.

# vi/etc/security/limits.conf

*-Nofile 102400

after the save exits , the maximum file descriptor has been permanently changed after logging back in.

the File-max parameter corresponds to the File-nr, which is read-only and allows you to view the current file descriptor usage.

# Sysctl-a|grep File

Fs.file-nr = 12800 0 782554

Fs.file-max = 782554

in theKernel 2.6in the previous version, File-nrthe values in are made up of three parts,respectively for: 1.number of file handles already allocated, 2.number of file handles that have not been assigned to a single use, 3.The maximum number of file handles. But inKernel 2.6the value of the second item in the version is always0,This is not a mistake .,It actually means that the file handle that has been allocated has been used without a waste .

the value of File-max is the maximum number of file handles that the Linux kernel can allocate. If you see a lot of error messages about the number of open files that have reached the maximum , You can try to increase the limit for that value. The default value of File-max is probably the 10% of the system memory

( system memory is calculated in kilobytes )

This article from the "11841085" blog, reproduced please contact the author!

MFS theory Knowledge