Ceph Cluster Expansion and Ceph Cluster Expansion
The previous article describes how to create a cluster with the following structure. This article describes how to expand the cluster.
IP |
Hostname |
Description |
192.168.40.106 |
Dataprovider |
Deployment Management Node |
192.168.40.107 |
Mdsnode |
MON Node |
192.168.40.108 |
Osdnode1 |
OSD Node |
192.168.40.148 |
Osdnode2 |
OSD Node |
Extensions mainly include the following:
- Add OSD process and CephMetadata service to mdsnode
- Add the Ceph Monitor service to osdnode1 and osdnode2
- Add a new OSD node osdnode3
The final result is as follows:
IP |
Hostname |
Description |
192.168.40.106 |
Dataprovider |
Deployment Management Node |
192.168.40.107 |
Mdsnode |
MDS, MON node, OSD Node |
192.168.40.108 |
Osdnode1 |
OSD node, MON Node |
192.168.40.148 |
Osdnode2 |
OSD node, MON Node |
192.168.40.125 |
Osdnode3 |
OSD Node |
Expand the OSD function of the MON Node
- Switch to the leadorceph user on dataprovider and enter the/home/leadorceph/my-cluster directory.
- OSD service for adding mdsnode nodes
ssh node1 sudo mkdir /var/local/osd2 exit
- Use the ceph-deploy command to create osd.
ceph-deploy --overwrite-conf osd prepare mdsnode:/var/local/osd2
ceph-deploy osd activate mdsnode:/var/local/osd2
After executing the preceding command, Ceph will re-adjust the entire cluster and migrate PG to the new OSD. You can use ceph-w to find that the cluster status has changed.
Add a new OSD Node
After the OSD function is added to an existing MON node, the OSD function will be added to a new osdnode3.
- Modify the/etc/hosts information of the dataprovider Node
Add192.168.40.125 osdnode3To its/etc/hosts file
- Run the following commands on osdnode3. For details, refer to "Building a Ceph storage cluster under Centos6.5"
yum install -y ntp ntpdate ntp-doc yum install -y openssh-server sudo useradd -d /home/leadorceph -m leadorceph sudo passwd leadorceph echo "leadorceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/leadorceph sudo chmod 0440 /etc/sudoers.d/leadorceph
Use the leadorceph user identity to execute the sudo mongodo command, and then modify Defaults requiretty to ults: ceph! Requiretty
- Copy the password-free SSH key to osdnode3 on the dataprovider Node
ssh-copy-id leadorceph@osdnode3
- Install osdnode3 ceph on dataprovider
ceph-deploy install osdnode3<br style="background-color: inherit;" />
- Create and activate processes on osdnode3 on dataprovider
ceph-deploy osd prepare osdnode3:/var/local/osd3ceph-deploy osd activate osdnode3:/var/local/osd3
It can be seen that the number of pg is too small at this time, and each OSD has only 16 on average, so you need to add some pg
ceph osd pool set rbd pg_num 100 ceph osd pool set rbd pgp_num 100
Add a Metadata Server
To use CephFS, you must include a metadata server to create the metadata service in the following ways:
ceph-deploy mds create mdsnode
Add Ceph Mon
The extension of Mon is complicated, which may lead to errors in the entire cluster. Therefore, we recommend that you do not modify MON at the beginning.
The number of ceph monitors is 2n + 1 (n> = 0) (ODD), and at least three are online, as long as the number of normal nodes> = n + 1, the ceph paxos algorithm ensures the normal operation of the system. Therefore, only one of the three nodes can be attached at the same time.
- Modify the ceph. conf file on dataprovider and append the information of the MON node to be added. The modification is as follows:
[global] auth_service_required = cephx osd_pool_default_size = 2 filestore_xattr_use_omap = true auth_client_required = cephx auth_cluster_required = cephx public_network = 192.168.40.0/24 mon_initial_members = mdsnode,osdnode1,osdnode2 fsid = 1fffdcba-538f-4f0d-b077-b916203ff698 [mon] mon_addr = 192.168.40.107:6789,192.168.40.108:6789,192.168.40.148:6789 mon_host = mdsnode,osdnode1,osdnode2 debug_mon = 20 [mon.a] host = mdsnode addr = 192.168.40.107:6789 [mon.b] host = osdnode1 addr = 192.168.40.108:6789 [mon.c] host = osdnode2 addr = 192.168.40.148:6789
- Run ceph-deploy mon create osdnode1 osdnode2
After adding a new node, You can query the result in the following method:
ceph quorum_status --format json-pretty
Storage and retrieval objects
The Ceph client uses the latest cluster map and CRUSH algorithm to determine how to map an object to PG, and then how to map the PG to OSD.
echo {Test-data} > testfile.txtrados put test-object-1 testfile.txt --pool=data
- Check whether the object is successfully added
rados -p data ls
ceph osd map data test-object-1
The output information may be as follows:
osdmap e537 pool 'data' (0) object 'test-object-1' -> pg 0.d1743484 (0.4) -> up [1,0] acting [1,0]
rados rm test-object-1 --pool=data
The location information may change dynamically due to changes in cluster nodes.