Shell Programming Application: Automated website backup and shell Automation
Rsync is a data image backup tool in unix-like systems-remote sync. A Fast Incremental Backup Tool, Remote Sync, supports local replication or synchronization with other SSH and rsync hosts. Compared with traditional cp, scp, and tar backup methods, rsync has the advantages of high security, fast backup, and support for Incremental backup. rsync can meet the demand for data backup with low real-time performance, for example, regularly back up the data on the file server to the remote server, and regularly perform data mirroring on the local disk.
Environment
Centos 5.8 (64) 192.168.23.130 (service)
Centos 5.8 (64) 192.168.23.131 (client)
Software
Rsync-3.1.1.tar.gz (http://rsync.samba.org /)
Installation Steps
1. Prerequisites
A. Disable selinux and firewall (in my personal habits, selinux and firewall must be disabled in whatever circumstances)
B. yum install-y make gcc
2. Install rsync (service, client)
A.tar-zxf rsync-3.1.1.tar.gz
B. cd rsync-3.1.1
C ../configure -- prefix =/usr/local/rsync
D. make
E. make install
3. Configure rsync
A. vim/etc/rsyncd. conf (service)
Uid = root
Gid = root
Use chroot = no
[Apps]
Auth users = root
Secrets file =/etc/rsyncd. passwd
Munge symlinks = no
Read only = no
Path =/data/test
B. vim/etc/rsyncd. passwd (service)
Root: 111111
C. vim/etc/rsync. passwd (client)
111111
4. Start rsync (service)
/Usr/local/rsync/bin/rsync -- daemon -- config =/etc/rsyncd. conf
5. Synchronize files (client)
/Usr/local/rsync/bin/rsync-vzrtopg -- progress -- delete rsync: // root@192.168.23.131/apps/tmp/-- password-file =/etc/rsync. pas (service-> client)
/Usr/local/rsync/bin/rsync-vzrtopg -- progress -- delete/tmp/rsync: // root@192.168.23.131/apps -- password-file =/etc/rsync. pas (client-> service)
6. Verify
Check whether synchronization files exist in the corresponding directory.
It will be run manually in the future, which is definitely not suitable for lazy people. Next we will introduce Inotify-tools to achieve real-time automatic synchronization. Inotify is a Linux feature that monitors file system operations, such as reading, writing, and creating. Inotify is responsive, easy to use, and much more efficient than the busy polling of cron tasks. Learn how to integrate inotify into your application and discover a group of command line tools that can be used to further automate system governance.
7. Install Inotify-tools
A. wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
B .tar-zxf inotify-tools-3.14.tar.gz
C. cd inotify-tools-3.14
D ../configure -- prefix =/usr/local/inotify
E. make
F. make install
8. Write shell scripts (rsync. sh)
#!/bin/sh #local dir dstdir="/tmp/tmp/" #sync user rsyncuser="root" #sync password rsyncpassword="/etc/rsync.pas" #remote ip remoteip="192.168.23.131" #remote module module="apps" #sync remote server module to local dir for ip in $remoteip do /usr/local/rsync/bin/rsync -vzrtopg --progress --delete $rsyncuser@$ip::$module $dstdir --password-file=$rsyncpassword done #monitor local dir, then rsync remote server module /usr/local/inotify/bin/inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f%e' -e close_write,modify,delete,create,attrib,move $dstdir | while read file do for ip in $remoteip do /usr/local/rsync/bin/rsync -vzrtopg --progress --delete $dstdir $rsyncuser@$ip::$module --password-file=$rsyncpassword echo " ${file} was rsynced" >> /tmp/rsync.log 2>&1 done done
9. chmod 755 rsync. sh
10. Run the shell script and check whether the operation is successful in the corresponding directory.
11. Set auto-start upon startup
Echo "sh/usr/local/inotify/rsync. sh &">/etc/rc. d/rc. local
Divergence: if there are multiple web servers online, we can use rsync + Inotify to upload files in batches. The specific method is to determine a server as the content distributor. All online servers use the rsync + Inotify method to monitor the content distributor. Once there is an update, the server is synchronized to the online in real time, this is also a small automated O & M.
Command parameter description
1. Details about rsyncd. conf
Global Parameters
Uid = root // the user who runs the RSYNC daemon
Gid = root // the group that runs the RSYNC daemon
Use chroot = no // do not use chroot
Max connections = 4 // The maximum number of connections is 4
Strict modes = yes // check the permission of the password file
Port = 873 // default port 873
Module Parameters
[Backup] // The Authentication Module name. You must specify
Path =/home/backup // the directory to which the image is to be created. It is indispensable!
Comment = This is a test // comments of This module
Ignore errors // ignore irrelevant IO errors
Read only = yes // read-only
List = no // column files are not allowed
Auth users = hening // The authenticated user name. If this row is not used, it indicates that the user is anonymous and has nothing to do with the system.
Secrets file =/etc/rsync. pas // table of password and user name comparison. The password file is generated by yourself.
Hosts allow = 192.168.1.1, 10.10.10.10 // allow hosts
Hosts deny = 0.0.0.0/0 // disable host
# Transfer logging = yes
Note: The following green files are automatically generated after the RSYNC service is installed.
Pid file =/var/run/rsyncd. pid // storage location of the pid file
Lock file =/var/run/rsync. lock // location of the lock file
Log file =/var/log/rsyncd. log // location where the log file is stored
2. rsync Command Parameters
-V, -- verbose detailed mode output
-Q, -- quiet simplified output mode
-C, -- checksum: enable the verification switch to force file transfer verification
-A, -- archive mode, indicating that the file is transmitted recursively and all file attributes are kept, equal to-rlptgoD.
-R, -- recursive processes subdirectories in recursive Mode
-R, -- relative uses relative path information
-B, -- backup creates a backup, that is, if the object already has the same file name, rename the old file ~ Filename. You can use the -- suffix option to specify different backup file prefixes.
-- Backup-dir: backs up files (for example ~ Filename) is stored in the directory.
-Suffix = SUFFIX defines the backup file prefix.
-U, -- update only performs updates, that is, skipping all files that already exist in DST and whose file time is later than the time to be backed up. (Do not overwrite the updated file)
-L, -- links retains soft links
-L, -- copy-links: process soft links like regular files
-- Copy-unsafe-links: only copies links other than the SRC path directory tree.
-- Safe-links ignores links other than the SRC path directory tree
-H, -- hard-links
-P, -- perms to keep File Permissions
-O, -- owner keeps file owner information
-G, -- group: Keep file group information
-D, -- devices: Keep Device File Information
-T, -- times preserve the file time information
-S, -- sparse performs special processing on sparse files to save DST Space
-N, -- dry-run which files will be transmitted
-W, -- whole-file: Copy files without incremental Detection
-X, -- one-file-system do not span the boundaries of the file system
-B, -- block-size = SIZE indicates the block size used by the algorithm. The default value is 700 bytes.
-E, -- rsh = COMMAND specifies that rsh and ssh are used for data synchronization.
-- Rsync-path = PATH specifies the path of the rsync command on the remote server
-C, -- cvs-exclude automatically ignores files in the same way as CVS to exclude files that do not want to be transmitted
-- Existing only updates the files that already exist in DST, instead of backing up the new files.
-- Delete: delete the files that are not in the SRC file in DST.
-- Delete-excluded: delete Files specified by this option at the receiving end.
-- Delete-after: delete after transmission
-- Ignore-errors is deleted when an IO error occurs in a timely manner.
-- Max-delete = NUM: a maximum of NUM files can be deleted.
-- Partial retains the files that are not completely transferred for any reason, so as to speed up subsequent re-transmission.
-- Force directory deletion, even if not empty
-- Numeric-ids does not match the number user and group ID with the user name and group name.
-- Timeout = time ip timeout, in seconds
-I, -- ignore-times do not skip files with the same time and length
-- Size-only: when determining whether to back up a file, only check the file size, regardless of the file time
-- Modify-window = NUM determines whether the timestamp window of the file is used at the same time. The default value is 0.
-T -- temp-dir = DIR create a temporary file in DIR
-- Compare-dest = DIR: compare the files in DIR to determine whether to back up data.
-P is equivalent to -- partial
-- Progress displays the backup process
-Z, -- compress compresses backup files during transmission
-- Exclude = PATTERN specifies to exclude file modes that do not need to be transmitted
-- Include = PATTERN specifies the file mode to be transmitted without exclusion
-- Exclude-from = FILE: exclude files in the specified mode in the FILE.
-- Include-from = FILE: files with the specified FILE pattern matching are not excluded.
-- Version: prints version information.
-- Address: bind to a specific address
-- Config = FILE: specify other configuration files. The default rsyncd. conf FILE is not used.
-- Port = PORT specify other rsync service ports
-- Blocking-io: block IO for remote shell
-Stats indicates the transmission status of some files.
-- SS actual transmission process during transmission
-- Log-format = formAT specifies the log file format
-- Password-file = FILE: Obtain the password permission 600 from the FILE.
-- Bwlimit = KBPS limits I/O bandwidth, KBytes per second
Inotifywait Parameters
-M: Keep listening
-R is used to recursively view directories.
-Q indicates the output event.
-E create, move, delete, modify, and attrib refer to the "listening for creating a mobile delete write permission" event.
