Build a Ceph storage cluster under Centos6.5

• Brief Introduction

The deployment mode of Ceph mainly includes the following types of nodes:
? Ceph OSDs: A Ceph OSD process is mainly used to store data, process data replication, recovery, and fill, adjust the resource combination and provide some monitoring information to Ceph Monitors by checking the heartbeat information of other OSD processes. when the Ceph Storage Cluster needs to prepare two data backups, at least two Ceph OSD processes must be in the active + clean state (Ceph will provide two data backups by default ).

? Monitors: Ceph Monitor maintains the cluster map status, including monitor map, OSD map, Placement Group (PG) map, and CRUSH map. ceph maintains Ceph Monitors, Ceph OSD Daemons, and historical records of PGs status changes (called an "epoch ").
? MDSs: Metadata stored by Ceph Metadata Server (MDS) represents the Ceph file system (I. e ., ceph Block Devices and Ceph Object Storage are not applicable to MDS ). ceph Metadata Servers allows system users to execute basic commands of POSIX file systems, such as ls and find.

• Hardware selection

CPUCeph metadata server dynamically distributes loads, which is CPU-sensitive. Therefore, Metadata Server should have better processor performance (such as quad-core CPU ). when Ceph OSDs runs the RADOS service, CRUSH is required to calculate the data storage location, replicate data, and maintain the copy of Cluster Map, therefore, OSD also needs proper processing performance. Ceph Monitors simply maintains the master information of Cluster Map, so this is not CPU-sensitive.

RAMMetadata servers and Monitors must be able to quickly provide data, so they must have sufficient memory (e.g ., 1 GB of RAM per daemon instance ). OSDs does not require too much memory for normal operations (e.g ., 500 MB of RAM per daemon instance); however, a large amount of memory (e.g ., ~ 1 GB per 1 TB of storage per daemon). Generally, and the more the better.

Data Storage
When planning data storage, consider the cost and performance trade-offs. At the same time, operating system operations, and multiple background programs to read and write a single drive will significantly reduce performance. There are also restrictions on the file system: BTRFS is not very stable for the production environment, but has the ability to record journal and write data in parallel, and XFS and EXT4 will be better.

Other considerations
You can run multiple OSD processes on each host, but you should ensure that the total throughput of the OSD hard disk does not exceed the network bandwidth required by the client to read or write data. You should also consider the data storage rate on each host. If the percentage on a specific host is large, it may cause a problem: to prevent data loss, it will cause Ceph to stop the operation.
When you run multiple OSD processes on each host, you also need to ensure that the kernel is up to date. When you run multiple OSD processes on each host (for example,> 20), many threads are generated, especially for recovery and relalancing operations. Many Linux kernels have a small default thread limit (for example, 32 K ). If you encounter this problem, consider setting kernel. pid_max a little higher. In theory, the maximum value is 4,194,303.

Networks
We recommend that each host have at least two 1 Gbps network interface controllers (NICS ). Since the throughput of most hard drive is about 100 MB/s, your Nic should be able to handle disk traffic on the OSD host. We recommend that you use at least two NICs as the public network and one as the cluster network. The cluster network (preferably not connected to the Internet) processes data replication and helps prevent denial-of-service attacks (preventing PG in active + clean status from replicating data across clusters ). Consider starting to use a 10 Gbps network in your rack. It takes 3 hours to Copy 1 TB of data over a 1 Gbps network, and 9 hours to copy 3 TBS (a typical drive configuration. In contrast, in a 10 Gbps network, the replication time is 20 minutes to 1 hour.

• Deployment

Deployed on four nodes

IP	Hostname	Description
192.168.40.106	Dataprovider	Deployment Management node
192.168.40.107	Mdsnode	MDS, MON node
192.168.40.108	Osdnode1	OSD node
192.168.40.148	Osdnode2	OSD node

• • Ceph deployment node
  • • Edit repo file

Sudo vim/etc/yum. repos. d/ceph. repo

[Ceph-noarch] name = Ceph noarch packages baseurl = http://ceph.com/rpm-performanceph-release#/?distro#/noarch enabled = 1 gpgcheck = 1 type = rpm-md gpgkey = https://ceph.com/git? P = ceph. git; a = blob_plain; f = keys/release. asc

• • • Install ceph-deploy

Sudo yum install ceph-deploy

• • • Edit/etc/hosts

Change the following content to the/etc/hosts file:

#127.0.0.1 localhost. localdomain localhost4 localhost4.localdomain4 127.0.0.1 dataprovider 192.168.40.107 mdsnode 192.168.40.108 osdnode1 192.168.40.148 osdnode2: 1 localhost. localdomain localhost6 extends

• • Ceph Node Setup

Each node executes the following command

Yum install-y ntp ntpdate ntp-doc yum install-y openssh-server

• • • Create a dedicated user

For ease of use, create a leadorceph user on each node and set the password to leadorceph.

Sudo useradd-d/home/leadorceph-m leadorceph sudo passwd leadorceph

• • • Add root permissions to users on each Ceph node

Echo "leadorceph ALL = (root) NOPASSWD: ALL" | sudo tee/etc/sudoers. d/leadorceph sudo chmod 0440/etc/sudoers. d/leadorceph

If the configuration is correct, the following error occurs:

• • Enable Password-less SSH

Switch to the leadorceph user on the deploy Node, generate an SSH key without a password, and copy it to each node.

[[Email protected] root] $ ssh-keygen Generating public/private rsa key pair. enter file in which to save the key (/home/leadorceph /. ssh/id_rsa): Created directory '/home/leadorceph /. ssh '. enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in/home/leadorceph /. ssh/id_rsa. your public key has been saved in/home/leadorceph /. ssh/id_rsa.pub. the key fingerprint is: a8: fb: b8: 3a: bb: 62: dc: 25: a6: 58: 7b: 9d: 97: 00: 30: ec [email protected] The key's randomart image is: + -- [RSA 2048] ---- + |. | + |. o | E .. | .. S |. o. | o. +. +. o. | o + o .. oo o | .. + * + o .. | + --------------- +

Copy to each node

Ssh-copy-id [email protected] ssh-copy-id [email protected] ssh-copy-id [email protected]

• • Storage Cluster
  • • Create a cluster

Create a configuration directory on the deploy Node. All operations on the following deploy nodes are performed in this directory.

Mkdir my-cluster cd my-cluster

• • • Deployment failed

If the deployment fails, run the following command to restore it:

Ceph-deploy purgedata {ceph-node} [{ceph-node}] ceph-deploy forgetkeys

• • • Create a cluster

Use ceph-deploy on the admin node to create a cluster, followed by the hostname of the mds node.

Ceph-deploy new mdsnode

After the execution is successful, three files will be added to the directory.

• • • Change the default number of replicas to 2

Modify ceph. conf to set the value of osd_pool_default_size to 2.

• • • Start Installation

Ceph-deploy install deploynode mdsnode osdnode1 osdnode2

Installed successfully:

• • Create a Ceph MON cluster monitor

Execute the following commands on deploy in sequence

Ceph-deploy mon create-initial ceph-deploy mon create mdsnode ceph-deploy gatherkeys mdsnode

After the execution is successful, the following files will appear

• • Create an OSD node

To achieve quick demonstration, the OSD background process uses directories instead of independent disks to create directories on two osd nodes respectively.

Ssh osdnode1 sudo mkdir/var/local/osd0 exit ssh osdnode2 sudo mkdir/var/local/osd1 exit

• • • Prepare an osd node

Ceph-deploy osd prepare osdnode1:/var/local/osd0 osdnode2:/var/local/osd1 ceph-deploy osd activate osdnode1:/var/local/osd0 osdnode2: /var/local/osd1

• • • Copy the configuration file and key file

Ceph-deploy admin dataprovider mdsnode osdnode1 osdnode2

• • • Ensure that ceph. client. admin. keyring. Have the correct permissions

      Sudo chmod + r/etc/ceph. client. admin. keyring

• • • View cluster status
      

• Problem summary
• • Question 1

Problem description: [ceph_deploy] [ERROR] RuntimeError: remote connection got closed, ensure ''requiretty ''is disabled for mdsnode

Solution: run the sudo mongodo command using the leadorceph user identity on the mdsnode, osdnode1, and osdnode2 nodes, and then modify

Defaults requiretty is Defaults: ceph! Requiretty

• • Problem 2 the following exception occurs when you execute the ceph-deploy admin command.
  • 
    

Solution

1. Modify the/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install. py file to cancel the install operation. The modification method is as follows:

2. Add the -- repo parameter during execution.

Build a Ceph storage cluster under Centos6.5