MONGODB Note 09

Backup

1. Backups are useful only if you are confident that you can quickly deploy in an emergency situation. So, regardless of which backup technology you choose, be sure to practice the backup and restore operations to know what to do.

2. Typically, the non-primary node of the replica set (relative to the master node) should be backed up.

3. Backing up the server

1). File system snapshot: Using a snapshot backup requires a journaling system to be turned on. If you are generating a snapshot of a running system, the data content of the snapshot is essentially equivalent to the data content that was forced to terminate with the kill-9 command. As a result, the Mongod will log on at startup

The file is replayed and then starts to run normally.

2). Copy the data file:

Method One:

A. Lock the database, prohibit any writes, and synchronize (Fsync) to flush all dirty pages to the hard disk to ensure that the files in the data directory are up-to-date and will not be changed. Db.fsynclock ()

B. When the Fsynclock command returns to the command line, copy all files in the data directory to the backup location. In Linux, you can use the following commands: Cp-r/data/db/*/mnt/external-drive/backup

C. After the data copy is complete, unlock the database so that it can write again: Db.fsyncunlock ()

Note that there are some locking issues with the authentication and Fsynclock commands. If authentication is enabled, do not close the shell during calls to Fsynclock () and Fsyncunlock (). If you disconnect during this period, you may not be able to reconnect and have to restart Mongod.

The Fsynclock () setting does not remain in effect after a reboot, Mongod always starts in a non-latching mode.

Method Two:

Close Mongod, copy the file, and then restart Mongod.

3). Recover Data Catalog Backup: Ensure that the Mongod is not running and that the directory to be recovered is empty. Copy the backed up data file to the data directory, and then restart Mongod. You can back up a separate database in this way as long as you know what files you want to copy.

4). Using Mongodump: Pros: You can back up individual databases, collections, and even subsets of a collection disadvantage: slow backup and recovery, and problems when working with replica sets.

If you want to run Mongod and mongodump on the same machine, you only need to specify the port that the Mongod runtime occupies: mongodump-p 31000 mongodump A dump directory is created in the current directory.

In any collection, if there is a unique index other than _id, you should refuse to use Mongodump and Mongorestore for backup.

4. Back up the replica set: Typically, backup nodes should be backed up. This reduces the burden on the primary node or locks the backup node without affecting the app (as long as the app does not send a read request to the backup node).

You can use one of the methods mentioned earlier to back up members of a replica set, but it is recommended to use a file system snapshot or copy a data file. There is no need to make any changes in either of these ways.

5. Backing up a shard cluster: In the face of a shard cluster, we are more concerned with chunking backup, which is to back up the configuration server and replica set separately. Before backing up and restoring a shard cluster, you should turn off the equalizer first. This is because you cannot get a consistent snapshot in an overly chaotic environment.

1). Back up and restore the entire cluster (not recommended):

2). Backing up and recovering individual shards: more often, you only need to restore a separate shard in the cluster. If you are not very picky, you can use the processing method just mentioned in the previous single server for backup recovery.

6. Use Mongooplog for incremental backups: we only need to make one backup and then use Oplog to back up all operations after that. This technique is more complex than the previous volume technology, so you should try to choose other technologies unless you really need them.

Deploying MongoDB

1. Design the system structure

1). Select storage Media: If you only consider performance, you should select it in order: memory, SSD, mechanical disk.

A. Due to the constraints, the general standard deployment scenario is to use less memory space and larger mechanical disk space. In this case, be aware that the working set size should be less than the memory capacity and should be prepared for device expansion when the working set grows.

B. Even faster mechanical hard drives do not make hard disk read time much shorter, so there is no need to spend too much time on such disks. More memory and SSD effects are better.

C. Normally we cannot add SSDs to an existing replica set if there is a mechanical hard disk in the replica set. If a machine that uses a solid-state drive becomes the primary member and takes over everything it can handle, the other members are limited by speed, unable to replicate the data in time,

In the back. Therefore, adding a new shard to a cluster is a better option if you are introducing a solid-state drive.

D. Consider using a mechanical hard drive to record the log, while recording the data with a solid-state drive.

2). Recommended RAID Configuration

A. RAID: Redundant array of independent disks, formerly redundant array of inexpensive disks, is a technique that allows us to use multiple disks as a single block disk. You can use it to improve the reliability or performance of your disks, or both.

A set of disks that use RAID technology is called a RAID disk array.

B. RAID10: Data is split to increase speed, and images are duplicated for increased reliability.

3). MongoDB has a very light load on the CPU. If you need to choose between memory and CPU for hardware investment, be sure to select memory. If you need to make a choice between speed and number of cores, choose the former. MongoDB can make better use of more cycles on a single processor than more parallel operations.

4). Select Operating System: 64-bit Linux operating system is the best choice to run on MongoDB. The Contos and Redhat Enterprise Editions may be the most common choice. You should use a stable version of the latest release, because older, defective software packages or cores can sometimes cause problems.

5). Swap space: A small swap space should be allocated to prevent excessive system memory usage, causing the kernel to terminate MongoDB operation. Then, MongoDB usually does not use any swap space.

Most of the memory used by MongoDB is "unstable": as long as the system requests memory space for some reason, the memory in memory is flushed to disk, and the meta-memory is replaced with other content. Therefore, the library data should never be written to the swap space,

Because it will first be flushed to disk. However, MongoDB uses the operation space when it needs to sort the data, that is, to index or sort operations. In doing so, MongoDB will try not to use too much memory, but if you do many of these operations at the same time,

The swap space is eventually used. If your application uses swap space on the server, you should try to redesign the application or reduce the load on that server.

5). File system: On Linux systems, it is recommended to use the EXT4 or XFS file system as the data volume. It is good to have a file system that can take a file system snapshot at backup time, but it can affect performance. Try to avoid using NFS file systems.

2. Virtualization: Using virtualization technology makes it easy to deploy systems with inexpensive hardware and to scale quickly. Then there are drawbacks to virtualization, especially the unpredictable network and disk IO conditions.

1). Disable memory over-allocation:

A. The setting of over-memory shards determines the policy that the current process should take when it requests too much memory from the operating system. Based on this setting, the kernel may allocate memory for the process, even if the memory is currently unavailable (the expected result is that the current process is using this memory

it becomes available). This kernel's memory behavior, which promises to the process not to exist, is called memory over-allocation. This feature allows MongoDB to not work well.

B. The value of Vm.overcommit_memory may be 0 (let the kernel guess the overallocated size), possibly 1 (satisfies all memory allocation requests), or 2 (the allocated virtual address space does not exceed the swap space and a small portion over-allocated and). Setting this value to 2 represents the

The most complex, but also the best choice. Run the following command to set this value to 2: $echo 2 >/proc/sys/vm/overcommit_memory no need to restart MongoDB after changing this setting

2). Mysterious Memory:

3). Handling IO issues with network disks

A. Do not host MongoDB in the cloud

B. Select an instance that can guarantee a certain amount of IOPS (Io Operations per Second, IO operations per second).

C. Monitor the volumes used by MongoDB. Once a volume slows down, it immediately terminates the run of the instance, and then launches an instance that uses another data volume. Monitoring content: Peak IO utilization ("IO latency" in MMS), loss of peak frequency, TCP packet growth,

The peak of the MongoDB read-write queue.

4). Using a non-network drive: A temporary drive is a disk that is physically connected to the machine key where the virtual machine (VM) resides, so there is no problem with many networked storage. These disks are temporary and cannot be used to hold important data.

3. System Configuration:

1). Disable NUMA: Disables every CPU access to all content in memory. MongoDB tends to access more data, even if it is inefficient, rather than accessing a small amount of data efficiently. Disabling NUMA is a magic button that boosts performance, so be sure to press it. It's like using a solid state drive

, disabling NUMA can improve the performance of everything.

A. If possible, disable NUMA by using BIOS disable. For example, if you are using grub, you can add the Numa=off option in grub.cfg: Kernel/boot/vmlinuz-2.6.38-8-generic root=/dev/sda ro Quiet Numa=off

B. If the system cannot disable NUMA in the BIOS, you can use the following options when you start Mongod: $ numactl--interleave=all mongod [options] adds this command to all used initialization scripts.

C. Disable the Zone-reclaim_mode option, which can be thought of as Super Numa, which moves the contents of memory back and forth to the local contents of the CPU: $ echo 0 >/proc/sys/vm/zone_reclaim_mode, No need to restart Mongod to take effect.

E. When NUMA is enabled, the MMS on the host is displayed in yellow. When NUMA is disabled, it turns blue.

2). Read data more intelligently: read-ahead is an optimization means that the operating system reads more data from the disk than it actually requests. The principle of this optimization is that most of the work done by the computer is continuous, that is, if the first 20M content of a video file is loaded, then it is likely to need

Use a few megabytes of content immediately following. As a result, more content is read from disk than is actually requested and put into memory for easy subsequent invocation.

A. MongoDB is not a typical workload, and setting up read-ahead is a common problem in MongoDB systems. MongoDB tends to randomly read a lot of small pieces of data from disk, so the default system settings do not work well. If there are too many pre-read content, the memory will gradually fill

MongoDB does not have the requested content, forcing MongoDB to access the disk more.

B. Use the Blockdev command to view the current read-ahead settings

C. You can change the read-ahead size setting through the--setra option: $ sduo blockdev--setra 16/DEV/SDB3 The recommended value is between 16 and 256.

The read-ahead size should not be set too small, otherwise reading a separate file requires multiple accesses to the disk. If the text is too large (greater than 1M), you should consider pre-reading more content. If the document is small, the pre-read values should be smaller, such as 32. Do not set the value of the read-ahead size even if the document is very small

Set to 16 or less. This can result in inefficient reading of index information.

D. You need to restart MongoDB for the read-ahead setting to take effect. This is because the process replicates a pre-read-size setting value at startup and continues to operate until the process stops running.

3). Disable large Memory pages: MongoDB needs to load a lot of small chunks of memory, so enabling large pages results in more disk IO.

4). Select a disk scheduling algorithm: After the disk controller receives the request from the operating system, a scheduling algorithm is used to determine the order in which the requests are processed. Sometimes changing this algorithm can improve disk performance.

Use the NoOp scheduling algorithm when using SSDs or using RAID controllers for caching.

You can use the--elevator option in a startup configuration to change the scheduling algorithm. Many times the scheduling algorithm can operate very well, the difference is not small.

5). Do not log access time: The system default record file last access time. Mongod operations are frequent, disabling this operation will improve performance. In a Linux system, you can change atime to Noatime in/etc/fstab, and record access time is forbidden.

6). Modification Restrictions: MongoDB may be affected by two restrictions: the number of threads that a process can establish and the number of processes that can open file descriptors (Files descriptor). Both should be set to unlimited.

4. Network Configuration

5. System Management:

1). Clock synchronization: Generally speaking, the clock error of each system is not more than one second is the most secure.

2). OOM Killer: If the MongoDB process is suddenly terminated and there is no error or exit information in the log, you should check/var/log/message (or another location where the kernel logs the content to see if there is information about terminating the Mongod process.

When the system has no swap space and the available memory starts to decrease, the OOM killer becomes particularly sensitive, so to avoid the hassle, you might want to configure the appropriate swap space. MongoDB should not use this swap space, but this will allow oom killer to relax.

If Oom killer terminates a mongos process, restart it.

3). Close Recurring tasks: Check for scheduled tasks or background processes that may be activated periodically and consume system resources, such as Automatic Updates to the package manager. These programs are activated and consume a lot of memory and CPU resources, and then disappear. We will not

Want to see these things on the production server.

To set the Swap space method steps:

1. Create Swapfile:

Under Root, create swapfile, assuming the current directory is "/", execute the following command:

# dd If=/dev/zero of=swapfile bs=1024 count=500000

Then in the root directory created a swapfile, the name "Swapfile", the size of 500M, you can also export files to any directory you want, the personal feel or directly placed in the root directory is better, Www.linuxidc.com at a glance, not easy to damage, put in other directories is not;

The options in the command explain:

---of: the path and name of the output interchange file;

---BS: block size, unit byte, generally 1k or 1024 byte;

---count: The total number of blocks is the total size of space, in blocks that are k;

---if: Read the source of free space, why is zero, not clear, first fixed so write it;

2. Set the Swapfile to swap space

# Mkswap Swapfile

3. Enable Swap space, this operation is somewhat similar to the mount operation (personal understanding):

# Swapon Swapfile

Now that the swap space has been increased, you can use the free command to see if the size of the swap space has changed;

4, if no longer use space can choose to turn off swap space, this operation is somewhat similar to the umount operation (Personal understanding)::

# Swapoff Swapfile

Using this method in each system startup requires manual setup, Swapfile, more trouble, workaround:

Append the following to the last line of the/etc/rc.d/rc.local file: (edit this file, of course, with VI ~)

/sbin/swapon/swapfile

Save and exit so that the swap space will automatically load after the system starts;

Summary: In the installation of the OS must be planned after the swap size, usually twice times the memory, but to consider the possibility of increasing memory later, so you can consider setting a little larger, but in our current widely used I386 PC, the maximum can not exceed 2G.

MONGODB Note 09