Tar Advanced Tutorial: Incremental backup, scheduled backup, network backup

I. Overview

Backup and recovery is critical to system maintenance. Unreasonable backups and restores can put your data at risk of loss. Many users are unaware of this risk when they lose important data. Recovering data from this situation will be time-consuming and difficult. So we should learn from the mistakes of others and make sure that your system is protected. Consider some questions first:

When do backups take place?

If you can accept one weeks of data loss, then a weekly backup is sufficient. But if you can only tolerate a day of data loss, then you have to make a backup every night.

Back to where?

Obviously backing up to the disk on your system is not a smart move, and users should back up to an external hard drive or tape.
To make the article look simpler, the examples in this article are not backed up to external devices, but in the/backup directory.

What do you want to back up?

We want to back up the entire system, so here's what we need to consider is a directory that doesn't have to be backed up :

• /backup
• /proc
• /lost+found
• /sys
• /mnt
• /media
• /dev
• /tmp

Backup method

Tar supports three ways of backing up:
1. Full backup: Back up all target files or folders specified
2. Incremental backup: Backup only atime files or folders that are different from the last backup
3. Differential backup: Similar to incremental backup, only adds support for Windows file system

This article mainly describes how to use tar to make incremental backups of the system and how to set up scheduled backups using Cron.

Second, the basic knowledgeMtime, Atime, CTime

The Linux file system will record the following three types of time
Mtime: File modification time. Updates when the contents of a file change. Ls-l Default Display Mtime
Atime: File access time. Updates automatically when a file is accessed. To view the atime of a file, perform a ls-lu.
　　　　/!\ Note:atime may change (in connection with the file system and the tar version) while the TAR is being backed up, and you can use –atime-preserve to prevent such updates.
CTime: File property modification time. Updated when file properties or permissions change. To view the atime of a file, perform a LS-LC.

Three, start incremental backup

GNU Tar uses the--listed-incremental option (abbreviated to-g) for incremental backups:

-g/backup/snapshot.snar --exclude=/backup--exclude=/proc--exclude=/lost+found--exclude=/sys--exclude=/ MNT--exclude=/media--exclude=/dev--exclude=/tmp/

Third, need to understand the

1. The last Slash "/" tells Tar to back up from the root directory.
2.tar relies heavily on atime, which can cause inconsistencies in the archive data files:
(1) Any access to files during backup
(2) Slow down the system time during backup
3.tar It is assumed that the disk device ID of the file to be backed up is constant, but some file systems such as NFS are not in this case and can be checked with--no-check-device blocking tar
4. Unless the situation is urgent and you fully trust tar, you may want to use the--verify option (abbreviated as-W) when backing up to ensure that the data is backed up correctly. Unfortunately, it does not support all storage media (such as tapes).

The difference between 5.–listed-incremental and –incremental (-G and-G)

–listed-incremental (-G) is a quick option for –listed-incremental (abbreviated to-G) and does not require a delta file to be specified. It is also a legacy of the old version of the tar incremental backup. It is now often used to decompress incremental backups, or list the files for incremental backups.
-G can create multi-level incremental backup files by copying Snar files, and-G can only be used with level0

Iv. List of backup files

TAR-TGVVF backup.tar.gz

-T equivalent to –list
-GVV a fixed usage that lists the properties of a file and also displays the backup properties:
Y – The file in the archive file
n – The file is not in the archive file
D – Folder

V. Data Recovery and restoration

Data recovery does not require that you specify an incremental file Snar for tar, because the information required for tar is already included in the packaged file.
To recover data, the following three ways are possible:

Method One:

Specify the delta file as/dev/null

Tar-xzvpf/backup/backup.tar.gz-g/dev/null-c/

Method Two:

Using the--incremental parameter (abbreviated to-G)

TAR-XZVPGF/BACKUP/BACKUP.TAR.GZ-C/

Method Three:

In some cases, you can also omit the-G option, but this is not the recommended version:

TAR-XZVPF/BACKUP/BACKUP.TAR.GZ-C/

Six, using cron scheduled backup

The new Cron Schedule task is simple and can be done in the following ways:

Method One:

$ crontab-e

Method Two:

# Vi/etc/crontab

On the author's Linux, method one can only run scheduled tasks with the privileges of the current user; method two specifies which user the program executes, but the file must be edited as root.
In this case, because you want to back up the entire file system, only/etc/crontab is described as an example:

# m H Dom Mon Dow user  Command 1 3 * * * root/root/scripts/daily_backup

This line of configuration indicates that the files in this script are executed 3:01/root/scripts/daily_backup per day.

Vii. DiscussionTar, cpio, rsync, dump

A previous backup was made using Cpio, but the cpio format was later found to be unable to extract individual files. But Tar also has its drawbacks:
1. It relies heavily on atime
2. If the archive file is restored to a new file system, all files Atime and CTime are refreshed. For a new system, the old incremental backup file is no longer applicable and must be done again.
In contrast, rsync seems to be a good solution to this problem, and it has a faster speed.
And dump is the best performance for file integrity in all popular backup software [3]

Viii. Script (scripts)Backup using the built-in incremental backup feature of tar

This code accomplishes the following tasks:

• Locate the Backup Database BKDB (record the level of the current backup), or start backup from level 0 if not
• Name the backup file and the Snar file at the current time and level
• Replicate older versions of Snar to boost level
• Assign a new Snar to tar for backup

/!\ Note: Please put this script in the/backup directory to run
[bash]
#!/bin/bash
# system Backup
# Sun Feb 00:34:42 CST 2012
# by Lesca

# Read db file and config
If [-F "bkdb"]; Then
Eval ' grep VER bkdb '
Eval ' grep NAME bkdb '
Old= $VER
new=$ ((old+1))
Else
New=0
Fi

Date= ' Date +%y%m%d-%h%m%s '
bkfile= $DATE-$NEW. tar.gz
newsnap= $DATE-$NEW. Snar
oldsnap= $NAME. Snar

# Create New Snapshot
If [-E "$OLDSNAP"]; Then
CP $OLDSNAP $NEWSNAP
Fi

tar-czvpf/backup/$BKFILE-G/backup/$NEWSNAP –exclude=/backup–exclude=/proc–exclude=/lost+found–exclude=/sys– Exclude=/mnt–exclude=/media–exclude=/dev–exclude=/tmp/

echo ver= $NEW > Bkdb
echo name= $DATE-$NEW >> bkdb
echo archived $BKFILE with snapshot $NEWSNAP
[/bash]

Recovering an incremental backup file

This code accomplishes the following tasks:

• Locate the Backup Database BKDB (record the level of the current backup), and if not, end
• Depending on the information specified by BKDB, locate the backup file and Snar to restore
• Checks if the folder that excludes the backup exists, does not exist, creates

/!\ Note: Restore refreshes atime and CTime
[bash]
#!/bin/bash

End () {
echo $
Exit
}

NameOf () {
ls-1 | grep $1.tar.gz | Cut-c1-15
}

If [-F "bkdb"]; Then
Eval ' grep VER bkdb '
If [$#-eq "1"] && [$1-le $VER]; Then
Ver=$1
Fi
Else
End "Error:no Backup archives."
Fi

Ver=0
While [$ver-le "$VER"]; Do
Archive= ' nameOf $ver '-$ver. tar.gz
Echo $ARCHIVE
TAR-XZVPGF $ARCHIVE-C/
ver=$ ((ver+1))
Done
Name= ' nameOf $VER '-$VER
echo ver= $VER > Bkdb
echo name= $NAME >> bkdb

if [!-e/proc]; Then
Mkdir/proc
Fi

if [!-e/lost+found]; Then
Mkdir/lost+found
Fi

if [!-e/sys]; Then
Mkdir/sys
Fi

if [!-e/mnt]; Then
Mkdir/mnt
Fi

if [!-e/media]; Then
Mkdir/media
Fi

if [!-e/dev]; Then
Mkdir/dev
Fi

if [!-e/tmp]; Then
Mkdir/tmp
Fi
[/bash]

Improved version: Use Find to back up only mtime updated files

This code [4] accomplishes the following tasks:

• Load current date to environment variable
• A full backup every Sunday to remove the previous incremental backup
• Incremental backups in peacetime

[bash]
#!/bin/bash
#
# creates backups of essential files
#
Data= "/HOME/ROOT/USR/LOCAL/HTTPD"
List= "/tmp/backlist_$$.txt"
#
Set $ (date)
#
if test "$" = "Sun"; Then
# Weekly A full backup of the all data and CONFIG. Settings
#
TAR-CZVF "/backup/data/data_full_$6-$2-$3.tgz" $DATA
Rm-f/backup/data/data_diff*
Else
# Incremental Backup:
#
Find $DATA-depth-type f \ (-ctime-1-o-mtime-1 \)-print > $LIST
TAR-CZTVF "backup/data/data_diff_$6-$2-$3.tgz" "$LIST"
Rm-f "$LIST"
Fi
[/bash]

Nine, Tar advanced tips1. File segmentation

Split at creation time:

Tar-cvpz <put Options Here>/| Split-d-B 3900m-/name/of/backup.tar.gz

Post-creation split:

Split-d-B 3900m/path/to/backup.tar.gz/name/of/backup.tar.gz

2. Network backup via Netcat

Send Side

TAR-CZVP <all other options>/| Nc-q 0 <receiving host> 1024

Receiving End

Nc-l 1024x768 > backup.tar.gz

/!\ Note : Because the parameter-F is no longer required to perform a backup archive name from a network backup, the name is specified by the receive-side NC redirection
{i} parameter description:
-Q 0:wait 0 second on QUIT
-L 1024:listen Port 1024

3. Network recovery via Netcat

Send Side

Cat Backup.tar.gz | Nc-q 0 <receiving host> 1024

Receiving End

Nc-l 1024 | TAR-XZVPF-C/

4. Network backup and recovery via SSH

On the host (SSH server) where the SSH service is installed, the client can complete the operation by simply initiating the connection and transmitting the command to the target computer (SSH server). All data is routed through the "Pipeline", and the user does not need to know which port to back up (and of course you can specify it yourself), and all transmitted data is encrypted. So with SSH backup, it is more secure, more convenient, and only one person can complete the backup work. Execute the following command on the computer where you are preparing the backup:

Tar-cvpz <all other options>/| SSH <backuphost> "(Cat > Ssh_backup.tar.gz)"

To recover a file, execute the following command:

Cat Ssh_backup.tar.gz | SSH <receivinng host> "(Tar-xzvp-c/)"

Tar Advanced Tutorial: Incremental backup, scheduled backup, network backup