Mysql High Availability DRBD + HEARTBEAT + MYSQL, drbdheartbeat

Mysql High Availability DRBD + HEARTBEAT + MYSQL, drbdheartbeat
1. Architecture Mysql:

Master <= slave

10.24.6.4: 3306 <= 10.24.6.6: 3306

VIP:

10.24.6.20

The VIP and mysql must be in the same network segment; otherwise, the ping fails.

 

2. References

Http://wenku.baidu.com/link? Url = QF1pG6DGMtl2uV1yt998xCduKsiEO7Bwn4CW79g0nx6EUw4iNgcBUz_jAA1W0FigPhjpiIs8FPLc9TxvFq_Yx7G-VvWeikFLzZchifM_tSS

3. DRBD shared memory settings

Shared Hard Disk/dev/sdb1

3.1. Create a New Hard Disk/dev/sdb1, but do not mount any directories.

 

 

3.2. Modify the hosts file and Host Name

On 10.24.6.4

Echo "10.24.6.4 drbd01">/etc/hosts

Echo "drbd01">/etc/hostname

On 10.24.6.6

Echo "10.24.6.6 drbd02">/etc/hosts

Echo "drbd02">/etc/hostname

3. Install drbd

On 10.24.6.4/10.24.6.6

Sudo apt-getinstall drbd8-utils

 

3. 4. Configure the configuration file of drbd. conf (/etc/drbd. conf)

Vim/etc/drbd. conf:

Include "/etc/drbd. d/global_common.conf ";

Include "/etc/drbd. d/*. res ";

In this way, you need to configure global_common.conf and create a resource file ending with res, Which is r0.res.

3.5. Configure global_common.conf (both master and slave are required)

Vim/etc/drbd. d/global_common.conf

 

Global {

Usage-count no; // help LINBIT calculate drbd usage}

Common {

Protocol C; // C is the safest and best-performing method to confirm the completion of write operations

Syncer {

Rate 100 M; // synchronization rate

}

}

. Configure ro. res (both master and slave nodes are required)

Sudo vim/etc/drbd. d/r0.res

Resource r0 {// r0 resource Name, which will be used in many places in the future

On drbd01 {

Device/dev/drbd0; // path of the Logical device

Disk/dev/sdb1; // physical device

Address 10.24.6.4: 7788; // master node

Meta-disk internal;

}

On drbd02 {

Device/dev/drbd0;

Disk/dev/sdb1;

Address 10.24.6.6: 7788; // slave Node

Meta-disk internal;

}

}

Annotations are not allowed.

3.7. initialize the metadata of drbd (operate on the master node)

10.24.6.4:

Sudo drbdadmcreate-md r0

 

3.8. handle errors

Run the following command: dd if =/dev/zero bs = 1 M count = 1 of =/dev/sdb1; sync

If you run drbdadm create-md r0 again and see success, the operation is successful.

3.9. Start the drbd Service (both master and slave nodes are executed)

/Etc/init. d/drbd start

 

Cat/proc/drbd

Ro: Secondary/Secondary indicates no primary disk has been created

Ds: Inconsistent/Inconsistent indicates that the data in the logical device (I .e./dev/drbd0) of the two nodes (master/slave) is not synchronized.

The hard disks of local and peer nodes may be in one of the following States:
Diskless: no local Block devices are allocated to DRBD. This indicates no available devices, or the drbdadm command is used for manual separation or the underlying I/O error causes automatic separation.
Attaching: the instantaneous status when no data is read.
Failed failure: the next status of the local block device that reports an I/O error. The next status is Diskless.
Negotiating: the instantaneous status before Attach reads data in the connected DRBD settings
Inconsistent: The data is Inconsistent, and a new resource is created immediately after the status appears on the two nodes (before the initial full synchronization. In addition, this status occurs on a node during synchronization (synchronization target ).
Outdated: The data resources are consistent, but they are Outdated.
DUnknown: this status occurs when the peer node's network connection is unavailable
Consistent: the data of an unconnected node is Consistent. When a connection is established, it determines whether the data is UpToDate or Outdated.
UpToDate: consistent latest data status, which is normal

3.10. Port Detection

10.24.6.4:

10.24.6.6:

 

3.11. initialize the network hard disk (executed by the master node)

Drbdadm -- overwrite-data-of-peer primary r0

 

View synchronization progress

Cat/proc/drbd

Ds: UpToDate/Inconsistent indicates that two nodes are synchronizing data.

3.12. Enable and disable resources (not required)

###### Manually enable resources
Drbdadm up <resource>
###### Manually disable Resources
Drbdadm down <resource>
Note:
Resource: indicates the resource Name. Of course, you can also use all to indicate [disabled | enabled] all resources.

3.13. Upgrade and downgrade resources (not required)

###### Upgrade Resources
Drbdadm primary <resource>
###### Downgrade a resource
Drbdadm secondary <resource>
Note: For DRBD in single-master mode, the two nodes are connected at the same time. Any node can be changed to the master node within a specific period of time, but only one of the two nodes can be the master node, if you already have a master, you need to downgrade it before you can upgrade it. This restriction is not imposed in dual-master mode.

3.14. How to view the resource connection status

[Root @ nod1 ~] # Drbdadm cstate r0 # r0 is the Resource Name
Connected

Status meaning:

The connection status of the resource. A resource may have one of the following connection statuses:
StandAlone independent: network configuration is unavailable; resources are not connected or managed to be disconnected (using the drbdadm disconnect command), or authentication failure or split-brain occurs.
Disconnecting disconnected: the disconnection is only a temporary state, and the next state is StandAlone.
Unconnected suspended: it is a temporary status before the connection attempt. The next status may be WFconnection and WFReportParams.
Timeout: the connection to the peer node times out and is also in the temporary status. The next status is Unconected.
BrokerPipe: the connection to the peer node is lost, which is also a temporary state. The next state is Unconected.
NetworkFailure: the temporary status after the connection is pushed to the peer node. The next status is Unconected.
ProtocolError: temporary status after the connection is pushed to the peer node. The next status is Unconected.
TearDown disassembly: temporary state. The peer node is closed, and the next state is Unconected.
WFConnection: waiting to establish a network connection with the peer node
WFReportParams: a TCP connection has been established. This node is waiting for the first network package from the peer node.
Connected connection: DRBD has established a connection. The data image is now available and the node is in normal state.
StartingSyncS: full synchronization. A synchronization has just started initiated by an administrator. In the future, the possible status will be SyncSource or PausedSyncS.
StartingSyncT: full synchronization. An administrator has just started synchronization and the next state is WFSyncUUID.
WFBitMapS: Some synchronization has just started. The next step may be in the SyncSource or PausedSyncS status.
WFBitMapT: Some synchronization has just started. The next step may be in the WFSyncUUID status.
WFSyncUUID: synchronization is about to begin. The next step may be SyncTarget or PausedSyncT.
SyncSource: synchronization on this node is in progress
SyncTarget: synchronization with the current node as the synchronization target is in progress
PausedSyncS: The local node is a continuous synchronization source, but the synchronization has been paused. It may be because another synchronization is in progress or the synchronization is paused using the command (drbdadm pause-sync ).
PausedSyncT: The local node is the target for continuous synchronization, but the synchronization has been paused. This can be because another synchronization is in progress or the synchronization is paused using the command (drbdadm pause-sync ).
VerifyS: Online device verification with local node as verification source is being executed
VerifyT: Online device verification with local nodes as the verification target is being executed

3.15. Resource synchronization completed

10.24.6.4:

10.24.6.6:

3.16. Data synchronization test

After formatting the disk, you can write data to it. The data is synchronized to the slave node.

 

Mkfs. ext4/dev/drbd0

Mount/dev/drbd0/mnt/data

(Note: only the primary disk node can be attached to the disk)

Cd/mnt/data

Mkdir testfile // create a test file named testfile

Umount/dev/drbd0 // detach a disk

Drbdadm secondary r0 // master downgrade, that is, the disk of the master node is the master disk at the beginning. Now it is downgraded to the slave disk, writing data to the master disk, and reading data from the slave disk. On drbd02 (slave node)

Drbdadm primary r0 // The master node is upgraded from the disk of the node to the master disk so that data on the disk can be read from the machine of the node.

Mount/dev/drbd0/mnt/data // after the master is upgraded, the permission cd/mnt/data ls-l will be available.

We will see the folder named testfile created in the master node. (For a moment:

To back up master-slave Mysql Data, we should be inspired here, I only need to save Mysql data to/mnt/data (we will put/dev/drbd0 IN THE PATH) so that the primary Mysql can write data to this region, drbd will automatically synchronize the data to the slave node.

Mysql on the server also reads data from here. In this way, two data copies are automatically backed up. )

 

 

3.17. Problem

Master downgrade failed:

 

Because the degraded directory is in use

4. MYSQL settings 4.1. Change the storage location of the Mysql database to a shared directory (executed on both the master and slave nodes)

By default, the storage location of the database is/var/lib/mysql (in/etc/mysql/my. the datadir location in cnf is found). Now you need to change it to exist in/share_mysql.

 

Copy data under/var/lib/mysql to/share_mysql

Cp-r/var/lib/mysql/share_mysql

Vim/etc/mysql/my. cnf

Change the value of datadir to/share_mysql/mysq.

4.2. Grant database users access permissions

 

4.3. System-level modification Permissions

140602 22:36:39 mysqld_safe Starting mysqld daemon withdatabases from/home/mysql

140602 22:36:39 [Warning] Can't create test file/home/mysql/localhost. lower-test

140602 22:36:39 [Warning] Can't create test file/home/mysql/localhost. lower-test

/Usr/libexec/mysqld: Can't change dir to '/home/mysql/' (Errcode: 13)

140602 22:36:39 [ERROR] Aborting

 

140602 22:36:39 [Note]/usr/libexec/mysqld: Shutdowncomplete

 

140602 22:36:39 mysqld_safe mysqld from pid file/var/run/mysqld. pid ended

 

You have modified the value of datadir in my. cnf.
You have already chown and chmod the owner and permissions of the new data directory or its parent path several times.
You have tried servicemysql start countless times, or/etc/init. d/mysql start, and mysql_install_db!

Congratulations! You saw this article. I found a solution after being stuck in the system for a few hours.

There are two reasons for this. Either of these causes will cause you to be notified of this warning by the system. If you are not a professional linux system security engineer or you are just a PHP programmer and have not studied system security in depth, you will not find the answer easily.

First, selinux, remember to install redhat on the Character interface when I was studying (a very old operating system ...) When there is such an option, we usually listen to the advice of the predecessors, change the default value to do not install it. However, if selinux is enabled on the machine that you want to operate on, it does not allow your mysql to operate on mysql_install_db at the new target location, and a warning is displayed in the title. A simple solution is to temporarily disable selinux using commands so that your operations can continue.
Setenforce 0
However, it is best to use a permanent method so that this product can be retained after restart.
Modify selinux = disabled in the/etc/SELINUX/config file, and restart or wait for the next restart.

Second, apparmor is the same as selinux. It also limits the directory permissions that mysql can use.
These two lines are available in the/etc/apparmor. d/usr. sbin. mysqld file, which specifies the data file path permission used by mysql.

/Var/lib/mysql/r,
/Var/lib/mysql/** rwk,

You must have seen that/var/lib/mysql/is the default path of the data file installed in mysql. apparmor controls the directory permissions that mysqld can use.
I want to move the data file to/data/mysql. In order to enable mysqld TO USE THE/data/mysql directory, add the following two files as shown above.

/Data/mysql/r,
/Data/mysql/** rwk,

Restart apparmor,/etc/init. d/apparmor restart
Then you can do what you want!

4.4. Database Synchronization Test

Start mysql on 10.24.6.4 to create a database named mongo_mysql.

The first step is to ensure that 10.24.6.4 is primary and 10.24.6.6 is secondary.

 

Master-slave switchover mounting

Service mysql stop # disable the mysql service

Drbdadm secondary r0 # downgrade 10.24.6.4 to backup

Umount/dev/drbd0 # detach a virtual hard disk

On drbd02

Drbdadm primary r0 # Upgrade 10.24.6.6

Mount/dev/drbd0 # mount a virtual hard disk

Service mysql start # start a database

Go to mysql to check whether a database named test has been synchronized from drbd01.

OK. Now the test is complete.

 

(Long-winded: A smart reader may ask, the efficiency of such operations is too low every time. Right, heartbeat will automatically help us do these things and it will listen to the service, once the service is stopped, the slave service will take over the service)

 

5. HEARTBEAT settings

Intelligent master-slave switchover for virtual hard disks

5.1. Install heartbeat

Apt-get install heartbeat

(Ps: The main configuration files of Heartbeat include ha. cf, haresources, and authkeys. After Heartbeat is installed, the three files are not available by default. You can download them from the official website or find them directly from the decompressed source code directory)

5.2. configuration file 5.3. Configure ha. cf.

Sudo cp/usr/share/doc/heartbeat/ha.cf.gz/etc/ha. d

Sudo gzip-d ha.cf.gz

Sudo vim ha. cf:

Logfacility local0 # This is the log for setting heartbeat. Here the system log is used.

Keepalive 500 ms # How long does it take to detect

Deadtime 10 # How long does it take to get in touch and think the other party has crashed (the unit is wonderful)

Warntime 5 # How long does it take to get in touch and start warning

Initdead 100 # Here is a period of ignore time reserved for the restart (for example, starting the network after the restart). If the network is not connected,

Keepalive detection is definitely not successful, but it cannot be switched at this time)

Bcast eth0

Auto_failback off # Whether to automatically switch back after recovery

Node drbd01 # node name (the Host Name of the machine in the cluster must be obtained through uname-n)

Node drbd02 # node name (it must be the slave machine name of the machine in the cluster, obtained through uname-n)

Ping 10.0.65.250 (Default Gateway, which can be obtained through the route command)

Respawn root/usr/lib/heartbeat/ipfail # Here is the function of configuring ip binding and switching. ipfail is the Program for controlling ip switching.

Apiauth ipfail gid = root uid = root # deadping 5, which is used to control ip address Switching

5.4. Configure the resource file/etc/ha. d/haresources

Sudo cp/usr/share/doc/heartbeat/haresources.gz/etc/ha. d/

Cd/etc/ha. d

Sudo gzip-d haresources.gz

The Haresources file is used to specify cluster resources such as the master node, cluster IP address, subnet mask, broadcast address, and startup service of the dual-machine system. Each row of the file can contain one or more resource script names, resources are separated by spaces, and parameters are separated by two colons. The files must be identical on the two HA nodes. The general format of this file is:

Node-name network <resource-group> node-name indicates the Host name of the master node, which must be consistent with the node name specified in the ha. cf file. Network is used to set the IP address, subnet mask, and network device ID of the cluster. Note that the IP address specified here is the IP address of the Cluster's external services. resource-group is used to specify the services to be managed by Heartbeat, that is, these services can be started and closed by Heartbeat. If you want to host these services, you must write the services as steps to start and close them through start/stop and put them in/etc/init. d/or/etc/ha. d/resource. in the d/directory, Heartbeat will automatically go to/etc/init according to the script name. d or/etc/ha. d/resource. d/directory.

The following describes the configuration methods:

Drbd01 IPaddr: 192.68.100.120/24/eth0: 0 drbddisk: r0

Filesystem:/dev/drbd0:/mnt/data: ext4 mysql

(Ps: in the preceding example, mysql resources are started and shut down. Therefore, you must go to/etc/ha. d/resource. d. Create a script to control the resource startup in this directory. The command is ln-s/etc/init. d/mysql/etc/ha. d/resource. d/mysql)

Drbd01 is the master node of the HA cluster, and IPaddr is an execution script that comes with heartbeat. Heartbeat will first execute/etc/ha. d/resource. d/IPaddr 192.68.100.120/24 start operation, that is, a virtual IP address with a subnet mask of 255.255.255.0 and an IP address of 192.68.100.120. This IP address is the network address that Heartbeat provides external services, and the network interface used by this IP address is also specified as eth0. Then, Heartbeat will mount the shared disk partition. "Filesystem:/dev/drbd0:/mnt/data: ext4" is equivalent to executing the mount operation under the command line, that is, "mount-t ext4/dev/drbd0/mnt/data", and finally mysql service. Note that the source file haresources must be identical between the master node and the backup node. Therefore, we should write

Drbd01 IPaddr: 192.68.100.120/24/eth0: 0

Drbddisk: r0

Filesystem:/dev/drbd0:/mnt/data: ext4 mysql cannot be written as follows:

Drbd01 drbddisk: r0 Filesystem:/dev/drbd0:/mnt/data: ext4 mysql192.68.100.120

Heartbeat Cannot use default route w/o netmask error will be reported, see: http://lists.linux-ha.org/pipermail/linux-ha/2002-November/005911.html

 

Drbd01 IPaddr: 10.24.6.20/24/eth0: 0 drbddisk: r0 Filesystem:/dev/drbd0:/share_mysql: ext4 mysql

 

 

5.5. Configure the resource file/etc/ha. d/authkeys

Sudo cp/usr/share/doc/heartbeat/authkeys/etc/ha. d/

The authkeys file is used to set the Heartbeat authentication method. There are three available authentication methods: crc, md5, and sha1. The security of the three authentication methods is increased in turn, but the system resources occupied are also increased in turn. If the Heartbeat cluster runs on a secure network, crc can be used. If the hardware configuration of each HA node is high, sha1 is recommended. This authentication method has the highest security level; if it is between network security and system resources, you can use the md5 authentication method. Here we use the crc authentication method and set it as follows: auth 1 1 crc #2 sha1 sha1_any_password #3 md5 md5_any_password

It should be noted that no matter what number is specified after auth, the next line must appear again as a key word. For example, if "auth 6" is specified ", there must be a "6 authentication type" line below ".

Finally, make sure that the permission for this file is 600 (that is,-rw ---).

　　

5.6. Start heartbeat5.7. start the master heartbeat

Startup log:

5.8. Start from heartbeat

Startup log

 

Master heartbeat log:

5.9. Verify 5.9.1. Master heartbeat port Detection

5.9.2. Detect from heartbeat Port

5.9.3. Master heartbeat service status

5.9.4. service status from heartbeat

 

5.9.5. Test the vip mysql Service

 

 

5.10. Mysql service switchover test 5.10.1. Switch over the master heaartbeat Service

5.10.2. Heartbea Service Transfer

5.10.3. Transfer Verification

Change the master drbd to drbd

From drbd to master drbd

The mysql service is not available on the master heartbeat machine.

Enable the mysql service from the heartbeat Machine

 

6. Summary

• Primary mysql and slave mysql can only have one service at the same time
• When the master mysql instance is down, heartbeat shares mysql Data through drbd and starts a new service from heartbeat.
• Master mysql and slave mysql can be transferred
• The master node normally occupies resources and runs all services. In case of a fault, the node is handed over to the slave node and the slave node runs the service.
• Suitable for dual-machine hot standby
• Share disk Architecture