在一台伺服器中以各資料庫的備份檔案為資料檔案啟動多個MySQL執行個體供SQL Review使用。
之前運行一直沒有問題(最多的時候有23個MySQL執行個體同時運行),後來新配置了一台伺服器,啟動其對應的執行個體時失敗。
部分錯誤記錄檔如下:
……
140505 16:05:59 InnoDB: Using Linux native AIO
140505 16:05:59 InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
InnoDB: Warning: io_setup() attempt 1 failed.
InnoDB: Warning: io_setup() attempt 2 failed.
InnoDB: Warning: io_setup() attempt 3 failed.
InnoDB: Warning: io_setup() attempt 4 failed.
InnoDB: Warning: io_setup() attempt 5 failed.
140505 16:06:02 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.
InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
140505 16:06:02 InnoDB: Fatal error: cannot initialize AIO sub-system
140505 16:06:02 [ERROR] Plugin 'InnoDB' init function returned error.
140505 16:06:02 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140505 16:06:02 [ERROR] Unknown/unsupported storage engine: InnoDB
……
通過錯誤記錄檔瞭解到最早發生錯誤的地方為 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts。
這個io_setup() failed with EAGAIN是關鍵。
我們man一下io_setup
NAME
io_setup - Create an asynchronous I/O context
……
DESCRIPTION
io_setup() creates an asynchronous I/O context capable of receiving at least maxevents. ctxp must not point to an AIO context that already exists, and must be
initialized to 0
prior to the call. On successful creation of the AIO context, *ctxp is filled in with the resulting handle.
RETURN VALUE
io_setup() returns 0 on success; otherwise, one of the errors listed in the "Errors" section is returned.
ERRORS
EINVAL ctxp is not initialized, or the specified maxevents exceeds internal limits. maxevents should be greater than 0.
EFAULT An invalid pointer is passed for ctxp.
ENOMEM Insufficient kernel resources are available.
EAGAIN The specified maxevents exceeds the user’s limit of available events.
ENOSYS io_setup() is not implemented on this architecture.
CONFORMING TO
……
看到io_setup用來建立非同步I/O上下文環境用於特定目的,錯誤碼EAGAIN意為指定的maxevents 超出了使用者可用events的限制。
該伺服器上已經運行了較多的MySQL執行個體,建立非同步I/O的資源已經達到了臨界,所以新的執行個體啟動失敗。
最後通過在啟動MySQL執行個體時加入 --innodb_use_native_aio = 0解決了問題。
也有通過更改系統設定來解決此問題的(待驗證)。
cat /proc/sys/fs/aio-max-nr可以查看到當前的aio-max-nr的值一般為65536(64k個)
可通過下述步驟改變該檔案中的值(上述檔案不能直接編輯)
sudo vim /etc/sysctl.conf
修改或加入
fs.aio-max-nr=262144(256k個)
執行命令修改/proc/sys/fs/aio-max-nr
sysctl -p
可以看到/proc/sys/fs/aio-max-nr中的值發生了變化
cat /proc/sys/fs/aio-max-nr
重啟MySQL執行個體
還有通過修改mysql源碼來避免該問題的,但一般情況下不推薦也沒有必要這麼做。