Openstack Cinder使用NetApp NFS作為後端儲存

標籤：python2.7   儲存   share   連結   toc   elf   ima   led   tor   

公司使用NetApp FS8020 R6作為測試環境NFS儲存使用。正好有機會測一下OpenStack的Cinder跟NetApp儲存整合。

說明：

1.OpenStack使用NetApp儲存直接掛載NFS檔案沒任何問題，生產中已使用比較穩定測試IOPS在160-220M/s。

2.使用OpenStack的Cinder無法像掛載如Linux主機共用NFS檔案那樣直接使用，需要調用NetAPP的API才能實現功能，如果設定為標準驅動現象為cinder-volume在開始的時候是正常的，但是一般十來分鐘後State狀態為down.(暫無)

設定檔如下:

[DEFAULT]
enabled_backends = nfs

[nfs]

volume_backend_name = nfs               //標黃的三處命名應統一，命名內容與使用協議無關如下文命名netapp_nfs
volume_driver = cinder.volume.drivers.nfs.NfsDriver    //定義使用的驅動類型，通用的NFS使用該選項，第三方廠商調用的驅動配置各不相同
nfs_sparsed_volumes = True
nfs_shares_config = /etc/cinder/nfs_shares
nfs_mount_point_base = $state_path/mnt
nfs_mount_options = v3

[[email protected] cinder]# vim nfs_shares

172.16.5.242:/vol/sqmgtvm02/nfs      //NetApp儲存IP:/共用的檔案目錄  正確的內容應為172.16.5.242:/vol/sqmgtvm02提供volume不是檔案夾nfs但是為了隔離生產中其他的檔案增加nfs（導致下文報錯2）

檢查/var/log/cinder/volume.log中報錯日誌如下:

2017-09-07 22:07:58.983 16612 ERROR oslo_service.service [req-37e3e47a-e1cb-47b8-a950-73374fd8713b - - - - -] Error starting thread.
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service Traceback (most recent call last):
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in run_service
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service service.start()
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/service.py", line 234, in start
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service self.manager.init_host(added_to_cluster=self.added_to_cluster)
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 425, in init_host
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service self.driver.init_capabilities()
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 704, in init_capabilities
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service stats = self.get_volume_stats(True)
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 512, in get_volume_stats
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service self._update_volume_stats()
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 448, in _update_volume_stats
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service provisioned_capacity = self._get_provisioned_capacity()
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 212, in _get_provisioned_capacity
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service run_as_root=True)
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service return processutils.execute(*cmd, **kwargs)
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389, in execute
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service cmd=sanitized_cmd)
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service ProcessExecutionError: Unexpected error while running command.
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du --bytes /var/lib/cinder/mnt/3d59744e62f876bf5171140e3a723d34
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service Exit code: 1
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service Stdout: u‘4096\t/var/lib/cinder/mnt/3d59744e62f876bf5171140e3a723d34/.snapshot\n8268\t/var/lib/cinder/mnt/3d59744e62f876bf5171140e3a723d34\n‘
2017-09-07 22:07:58.983 16612 ERROR oslo_service.service Stderr: ‘/bin/du: WARNING: Circular directory structure.\nThis almost certainly means that you have a corrupted file system.\nNOTIFY YOUR SYSTEM MANAGER.\nThe following directory is part of the cycle:\n \xe2\x80\x98/var/lib/cinder/mnt/3d59744e62f876bf5171140e3a723d34/.snapshot/sv_nightly.0\xe2\x80\x99\n\n‘

 

2017-09-09 21:33:28.066 154678 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-09-09 21:33:28.500 154678 WARNING cinder.keymgr.conf_key_mgr [req-ba7d370f-a96c-4b3f-95fa-c6234277766e - - - - -] This key manager is insecure and is not recommended for production deployments
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume [req-ba7d370f-a96c-4b3f-95fa-c6234277766e - - - - -] Volume service [email protected] failed to start.
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume cluster=cluster)
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/service.py", line 382, in create
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume cluster=cluster)
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/service.py", line 202, in __init__
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume *args, **kwargs)
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 242, in __init__
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume active_backend_id=curr_active_backend_id)
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume return import_class(import_str)(*args, **kwargs)
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/common.py", line 75, in __new__
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume na_utils.check_flags(NetAppDriver.REQUIRED_FLAGS, config)
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/utils.py", line 79, in check_flags
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume raise exception.InvalidInput(reason=msg)
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume InvalidInput: Invalid input received: Configuration value netapp_storage_protocol is not set.
2017-09-09 21:33:28.513 154678 ERROR cinder.cmd.volume
2017-09-09 21:33:28.517 154678 ERROR cinder.cmd.volume [req-ba7d370f-a96c-4b3f-95fa-c6234277766e - - - - -] No volume service(s) started successfully, terminating.
2017-09-09 21:33:30.401 154691 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-09-09 21:33:33.308 154691 WARNING cinder.keymgr.conf_key_mgr [req-44d8acf3-246c-4efb-aaaf-00d092a68f40 - - - - -] This key manager is insecure and is not recommended for production deployments

正確設定檔：

[netapp_nfs]
volume_backend_name = netpp_nfs
volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
netapp_storage_family = ontap_7mode          //NetApp目前產品線分為兩種模式
netapp_storage_protocol = nfs                        //使用協議
netapp_server_hostname = sqmgtvm02         //改名稱建議修改/etc/hosts來定義主機和IP (原使用共用目錄的172.16.X.X的IP地址提示認證錯誤更改為NetApp的管理地址正常)
netapp_server_port = 80                             
netapp_transport_type = http                          //認證協議選擇支援https和http，標準中我使用http模式。（https配置較複雜詳見下文參考連結）
netapp_login = root                                         //登入使用者名稱,應該為管理員權限，就是登陸onecommand的那個帳號和密碼
netapp_password = netappxxx                       //登入密碼
#netapp_vserver = svm_name                       //具體未知 按照官方文檔推測應該就是sqmgtvm02
nfs_shares_config = /etc/cinder/nfs_shares   //配置NetApp NFS儲存共用內容，可以使用showmount -e 172.16.5.xxx 來顯示儲存共用的目錄
nfs_mount_point_base = $state_path/mnt     //掛載到本地的掛載點，改命令直接掛載至/var/lib/cinder/mnt/6ff41da189e9ce5bfc54af3394adbcd8
#max_over_subscription_ratio = 1.0              //推測為磁碟超配比
#reserved_percentage = 5                             //卷預留空間佔比防止卷徹底掛掉，Ceph中也有類似選項避免空間爆掉可以通過釋放改空間來執行緊急刪除或遷移操作

 

 

 

 

Openstack Cinder使用NetApp NFS作為後端儲存