Build a Ceph storage cluster under Centos6.5
|
|
|
|
|
|
|
- Deployment Management Node
|
|
|
|
|
|
|
|
|
|
|
|
|
Brief Introduction
The deployment mode of Ceph mainly includes the following types of nodes:
? Ceph OSDs: A Ceph OSD process is mainly used to store data, process data replication, recovery, and fill, adjust the resource combination and provide some monitoring information to Ceph Monitors by checking the heartbeat information of other OSD processes. when the Ceph Storage Cluster needs to prepare two data backups, at least two Ceph OSD processes must be in the active + clean state (Ceph will provide two data backups by default ).
? Monitors: Ceph Monitor maintains the cluster map status, including monitor map, OSD map, Placement Group (PG) map, and CRUSH map. ceph maintains Ceph Monitors, Ceph OSD Daemons, and historical records of PGs status changes (called an "epoch ").
? MDSs: Metadata Stored by Ceph Metadata Server (MDS) represents the Ceph File System (I. e ., ceph Block Devices and Ceph Object Storage are not applicable to MDS ). ceph Metadata Servers allows system users to execute basic commands of POSIX file systems, such as ls and find.
Hardware Selection
CPUCeph metadata server dynamically distributes loads, which is CPU-sensitive. Therefore, Metadata Server should have better processor performance (such as quad-core CPU ). when Ceph OSDs runs the RADOS service, CRUSH is required to calculate the data storage location, replicate data, and maintain the copy of Cluster Map, therefore, OSD also needs proper processing performance. Ceph Monitors simply maintains the master information of Cluster Map, so this is not CPU-sensitive.
RAMMetadata servers and Monitors must be able to quickly provide data, so they must have sufficient memory (e.g ., 1 GB of RAM per daemon instance ). OSDs does not require too much memory for normal operations (e.g ., 500 MB of RAM per daemon instance); however, a large amount of memory (e.g ., ~ 1 GB per 1 TB of storage per daemon). Generally, and the more the better.
Data StorageWhen planning data storage, consider the cost and performance trade-offs. At the same time, operating system operations, and multiple background programs to read and write a single drive will significantly reduce performance. There are also restrictions on the file system: BTRFS is not very stable for the production environment, but has the ability to record journal and write data in parallel, and XFS and EXT4 will be better.
Other considerations
You can run multiple OSD processes on each host, but you should ensure that the total throughput of the OSD hard disk does not exceed the network bandwidth required by the client to read or write data. You should also consider the data storage rate on each host. If the percentage on a specific host is large, it may cause a problem: to prevent data loss, it will cause Ceph to stop the operation.
When you run multiple OSD processes on each host, you also need to ensure that the kernel is up to date.
When you run multiple OSD processes on each host (for example,> 20), many threads are generated, especially for recovery and relalancing operations. Many linux kernels have a small default thread limit (for example, 32 K ). If you encounter this problem, consider setting kernel. pid_max a little higher. In theory, the maximum value is 4,194,303.
NetworksWe recommend that each host have at least two 1 Gbps network interface controllers (NICS ). Since the throughput of most hard drive is about 100 Mb/s, your Nic should be able to handle disk traffic on the OSD host. We recommend that you use at least two NICs as the public network and one as the cluster network. The cluster network (preferably not connected to the Internet) processes data replication and helps prevent denial-of-service attacks (preventing PG in active + clean status from replicating data across clusters ). Consider starting to use a 10 Gbps network in your rack. It takes 3 hours to copy 1 TB of data over a 1 Gbps network, and 9 hours to copy 3 TBS (a typical drive configuration. In contrast, in a 10 Gbps network, the replication time is 20 minutes to 1 hour.
DeploymentDeployed on four nodes
Ceph deployment node editing repo file sudo vim/etc/yum. repos. d/ceph. repo
[ceph-noarch] name=Ceph noarch packages baseurl=http://ceph.com/rpm-{ceph-release}/{distro}/noarch enabled=1 gpgcheck=1 type=rpm-md gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.ascInstall ceph-deploy
sudo yum install ceph-deploy
Edit/etc/hosts
Change the following content to the/etc/hosts file:
#127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 127.0.0.1 dataprovider 192.168.40.107 mdsnode 192.168.40.108 osdnode1 192.168.40.148 osdnode2 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Ceph Node Setup each Node executes the following command
yum install -y ntp ntpdate ntp-doc yum install -y openssh-server
Create a dedicated user
For ease of use, create a leadorceph user on each node and set the password to leadorceph.
sudo useradd -d /home/leadorceph -m leadorceph sudo passwd leadorceph
Add root permissions to users on each Ceph Node
echo "leadorceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/leadorceph sudo chmod 0440 /etc/sudoers.d/leadorceph
If the configuration is correct, the following error occurs:
Enable Password-less SSH switch to leadorceph user on the deploy node, generate an SSH key without a Password, and copy it to each node.
[leadorceph@dataprovider root]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/leadorceph/.ssh/id_rsa): Created directory '/home/leadorceph/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/leadorceph/.ssh/id_rsa. Your public key has been saved in /home/leadorceph/.ssh/id_rsa.pub. The key fingerprint is: a8:fb:b8:3a:bb:62:dc:25:a6:58:7b:9d:97:00:30:ec leadorceph@dataprovider The key's randomart image is: +--[ RSA 2048]----+ | . | | + | | . o | | E . . | | .. S | | .o o. | |o.+.+. o . | |o+o..oo o | |..+*+o.. | +-----------------+
Copy to each node
ssh-copy-id leadorceph@mdsnode ssh-copy-id leadorceph@osdnode1 ssh-copy-id leadorceph@osdnode2
Create a storage Cluster
Create a configuration directory on the deploy node. All operations on the following deploy nodes are performed in this directory.
mkdir my-cluster cd my-cluster
Deployment failed
If the deployment fails, run the following command to restore it:
ceph-deploy purgedata {ceph-node} [{ceph-node}] ceph-deploy forgetkeysCreate a cluster
Use ceph-deploy on the admin node to create a cluster, followed by the hostname of the mds node.
ceph-deploy new mdsnode
After the execution is successful, three files will be added to the directory.
Change the default number of replicas to 2
Modify ceph. conf to set the value of osd_pool_default_size to 2.
Start Installation
ceph-deploy install deploynode mdsnode osdnode1 osdnode2
Installed successfully:
Create a Ceph MON cluster monitor and execute the following commands in sequence on deploy.
ceph-deploy mon create-initial ceph-deploy mon create mdsnode ceph-deploy gatherkeys mdsnode
After the execution is successful, the following files will appear
Create An OSD Node
To achieve quick demonstration, the OSD background process uses directories instead of Independent Disks to create directories on two osd nodes respectively.
ssh osdnode1 sudo mkdir /var/local/osd0 exit ssh osdnode2 sudo mkdir /var/local/osd1 exit
Prepare an osd Node
ceph-deploy osd prepare osdnode1:/var/local/osd0 osdnode2:/var/local/osd1 ceph-deploy osd activate osdnode1:/var/local/osd0 osdnode2:/var/local/osd1
Copy the configuration file and key file
ceph-deploy admin dataprovider mdsnode osdnode1 osdnode2
Ensure that ceph. client. admin. keyring. Have the correct permissions
sudo chmod +r /etc/ceph/ceph.client.admin.keyring
View Cluster status
Problem SummaryProblem 1 Problem description: [ceph_deploy] [ERROR] RuntimeError: remote connection got closed, ensure ''requiretty ''is disabled for mdsnode
Solution: Run The sudo mongodo command using the leadorceph user identity on the mdsnode, osdnode1, and osdnode2 nodes, and then modify
Defaults requiretty is Defaults: ceph! Requiretty
Problem 2 The following exception occurs when you execute the ceph-deploy admin command.
Solution
1. Modify the/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install. py file to cancel the install operation. The modification method is as follows:
2. Add the -- repo parameter during execution.
