Simple instance configuration for MogileFS

Simple instance configuration for MogileFS
Experimental principle: MogileFS is an open-source Distributed File System used to build a distributed file cluster developed by DangaInteractive under LiveJournal, the Danga team developed good open-source projects including Memcached, MogileFS, and Perlbal: (Note: Perlbal is a powerful reverse proxy server written in Perl ). MogileFS is an Open Source Distributed File System. Main features include: Application Layer components, no spof, automatic file replication, better reliability than RAID, no RAID support, etc ...... The core role is as follows: Tracker node: stores the metadata information of each node file in the database to save the storage location distribution of all keys in each domain, so that you can conveniently retrieve and locate the Data Location and monitor each node, inform the client of the location of the storage zone and command the storage node to copy data copies. The process name is mogilefsd (7001 ). Database node: provides data access services for tracker nodes. Storage node: converts keys in a specified domain to their unique file names and stores them in a specified device file. The converted file name is a value. The storage node automatically maintains the key-value relationship, because the storage node uses http for data transmission, it depends on perlbal. The storage node front-end can use nginx for reverse proxy, but the nginx-mogilefs-module-master module must be installed for name conversion, process name: mogstored (7501), perbal (7500 ). Domain: the key value in a Domain is unique. A MogileFS can have multiple domains, and the Domain can be used to store containers of different application types of data. Host: each storage node is called a Host. A Host can have multiple storage devices (separate hard disks). Each device has an ID, and Domain + Fid is used to locate files. Class: The minimum unit for copying, manages file attributes, and defines the number of copies of files stored on different devices. Experimental System: CentOS 6.6 _ x86_64 prerequisites: Prepare the compiling environment in advance. Firewall and selinux both disable the experiment. Note: There are three hosts in this experiment. IP address and role allocation are shown in topology: http://pan.baidu.com/s/1bnnYiMr experiment topology: 1. Install MySQL and MogileFS on 19.66. install mysql:

tar xf mariadb-10.0.20-linux-x86_64.tar.gz  -C /usr/local/cd /usr/local/ln -sv mariadb-10.0.20-linux-x86_64 mysqluseradd -r mysqlmkdir -pv /mydata/datachown -R mysql.mysql /mydata/data/cd mysql/chown -R root.mysql .scripts/mysql_install_db --user=mysql --datadir=/mydata/data/cp support-files/my-large.cnf /etc/my.cnfcp support-files/mysql.server /etc/init.d/mysqldchkconfig --add mysqldchkconfig mysqld on

 

2. Install YAML:

wget http://search.cpan.org/CPAN/authors/id/I/IN/INGY/YAML-1.15.tar.gztar xf YAML-1.15.tar.gzcd YAML-1.15perl Makefile.PLmake && make install

 

3. Use cpan to install MogileFS online and related modules: cpan ---------------------------------> install MogileFS: Server // enter yes If yes is entered. 4. Create an authorized user:

/usr/local/mysql/bin/mysql------------------------------------------>USE mysql;UPDATE user SET Password=PASSWORD(123456) WHERE User='root';GRANT ALL ON mogilefs.* TO 'moguser'@'192.168.19.%' IDENTIFIED BY '123456';GRANT ALL ON *.* TO 'root'@'192.168.19.%' IDENTIFIED BY '123456';FLUSH PRIVILEGES;

 

5. initialize the database: mogdbsetup -- dbhost = 192.168.19.66 -- dbrootuser = root -- dbrootpass = 123456 -- dbuser = moguser -- dbpass = 123456 -- yes if the following error occurs, run the command again, normal data can be imported normally: 6. create a Tracker configuration file:

mkdir /etc/mogilefsvim /etc/mogilefs/mogilefsd.conf--------------------------------------------------------->db_dsn = DBI:mysql:mogilefs:host=192.168.19.66db_user = moguserdb_pass = 123456conf_port = 7001listener_jobs = 5node_timeout = 5rebalance_ignore_missing = 1

 

7. Create a mogile user and start the test. Check the port and check that 7001 has started listening. You can also use "pkill mogilefsd" to stop the mogilefsd service. Useradd mogilesu mogile-c "mogilefsd-c/etc/mogilefs/mogilefsd. conf -- deamon "2. Install MogileFS 1 on 19.74 and 19.76. install YAML: wget http://search.cpan.org/CPAN/authors/id/ I /IN/INGY/YAML-1.15.tar.gztar xf YAML-1.15.tar.gzcd YAML-1.15perl Makefile. PLmake & make install2. use cpan to install MogileFS online: cpan --------------------------------> install MogileFS: Server3. create a Storage configuration file:

Mkdir/data/mogdata-pvuseradd mogilechown-R mogile. mogile/data/mogdatamkdir/etc/mogilefsvim/etc/mogilefs/mogstored. conf ---------------------------------------------------> httplisten = 192.168.19.74: 7500 // change the address on 19.76 to mgmtlisten = 192.168.19.74: 7501 docroot =/data/mogdata

 

4. Start the MogileFS Storage node and check the ports. The Listening Starts at 7500 and 7501. In addition, you can use "pkill mogstored" to stop the mogstored service. Mogstored -- daemon 3. Manage and configure MogileFS 1. install MogileFS-Utils: cpan ----------------------------------> install MogileFS: Utils2. Add a Storeage node to Tracker: mogadm -- trackers = 192.168.19.66: 7001 host add test -- ip = 192.168.19.74 -- port = 7500 -- status = alivemogadm -- trackers = 192.168.19.66: 7001 host add tsst -- ip = 192.168.19.76 -- port = 7500 -- status = alive3. create a storage directory: On instance 19.74: mkdir/data/mogdata/dev1: mkdir/data/mogdata/dev24. add the storage device: mogadm -- trackers = 192.168.19.66: 7001 device add test 1 mogadm -- trackers = 192.168.19.66: 7001 device add tsst 2 mogadm -- trackers = 192.168.19.66: 7001 device listmogadm -- trackers = 192.168.19.66: 7001 checkmogadm domain add jason 5. upload file test: mogupload -- trackers = 192.168.19.66: 7001 -- domain = jason -- key = '/fstab.html' -- file = '/etc/fstab' mogfileinfo -- trackers = 192.168.19.66: 7001 -- domain = jason -- key = '/fstab.html' Open the connection in the red box in a browser and you will see the file: Let's upload a picture and try: mogupload -- trackers = 192.168.19.66: 7001 -- domain = jason -- key = '/centos.png' -- file = '/usr/share/backgrounds/default.png' mogfileinfo -- trackers = 192.168.19.66: 7001 -- domain = jason -- key = '/centos.png'
4. Expand the experiment above. We can see that the uploaded files are only stored on one host. In fact, they should be stored on both nodes. The cause of the error may be caused by the Sys: Syscall module. We can see that the current version of this module is 0.25. I found that if this software is reduced to version 0.23 may fix this problem, let's try: wget http://search.cpan.org/CPAN/authors/id/ B /BR/BRADFITZ/Sys-Syscall-0.23.tar.gztar xf Sys-Syscall-0.23.tar.gzcd Sys-Syscall-0.23perl Makefile. PLmake & make install restart the mogilefsd process, and then upload a file. Try: vim/root/haha // create a file ----------------------------> 123 <using mogupload -- trackers = 192.168.19.66: 7001 -- domain = jason -- key = '/haha.html' -- file = '/root/hahaha' mogfileinfo -- trackers = 192.168.19.66: 7001 -- domain = jason -- key ='/haha.html'