Restore the MySQL-mmm Master from the REPLICATION_FAIL status

 

It is estimated that there are more than a hundred yuan to see. I always have to explain the background first.

A few days ago, the website suddenly failed to be accessed, and there was no content except the framework on the page. The following error messages are displayed in the system running log:

 

Communications link failure

 

The last packet successfully received ed from the server was 7,875,055 milliseconds ago. The last packet sent successfully to the server was 7,875,055 milliseconds ago.

 

The last sentence is:

 

<Span style = "color: # ff0000;"> Caused by: java. SQL. SQLException: The table 'message' is full </span>

This is incredible. When there are no current users, the database cannot be fully written. So I checked it on the database server Master1 and found that/var/is full by running the df-h command. Note: When the database was created, all the data files were stored in another directory, and then/var/lib/mysql/was softlink. In this case, the table was not moved to that directory after it was created. The next step is:

1. service mysql stop MySQL service

2. Move the data table file to the specified directory to create a softlink

3. service mysql start MySQL service

4. Go to MySQL-mmm through mmm_control set_offline db01, then mmm_control set_online db01, and re-launch master01.

Then you can view the status through mmm_control show, which is already ONLINE.

 

This is the end, NO! NO! This is not GC, according to the practice of baibai (I am advertising baibai, absolutely not.

When I heard a report today, I suddenly wanted to see the running status of MySQL-mmm. Mmm_control show: this is a scene that you don't want to see. The status of db01 is REPLICATION_FAIL, set_offline, set_online, And the MySQL service is restarted.

Go to db01 to view the error log and see the following information:

 

111104 13:19:19 [ERROR]/usr/sbin/mysqld: Table 'table1' is marked as crashed and shoshould be retried red

111104 13:19:19 [ERROR] Slave SQL: Error 'table 'table1' is marked as crashed and shoshould be retried red' on query. Default database: 'db1'. Query :'...'

111104 13:19:19 [Warning] Slave: Table './db1/table1' is marked as crashed and shoshould be retried red Error_code: 145

111104 13:19:19 [Warning] Slave: Table 'table1' is marked as crashed and shoshould be retried red Error_code: 1194

111104 13:19:19 [ERROR] Error running query, slave SQL thread aborted. fix the problem, and restart the slave SQL thread with "SLAVE START ". we stopped at log 'mysql-bin-master2.000022 'position 110544518

 

Log on to the database and run:

 

Mysql> repair table table1;

Mysql> start slave;

Check the error log and you will see:

 

111104 13:19:19 [Note] Slave I/O thread: connected to master 'replication @ db02: 8080', replication started in log 'mysql-bin-master2.000022 'at position 3306

111104 13:24:18 [Note] Found 11845 of 11846 rows when repairing './db1/table1'

111104 13:27:03 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin-master2.000022 'at position 110544518, relay log'/mysql/log_vol/replication/mysql-bin-master1.004525' position: 844646

 

Check the status on the MySQL-mmm monitoring server. You can see db01 from REPLICATION_FAIL to REPLICATION_DELAY to ONLINE. After waiting for a while, it's always ONLINE, and it seems stable. However, writer is still in db02. First, switch db02 set_offline and db02 set_online. You can see that writer switched to db01.

 

Is there GC? Haha, just solve the problem :-)

From mydeman's Learning Log