Ceph RBD CEPHFS Storage

Ceph RBD CEPHFS Storage

Environment Preparation: (here only do basic tests, Ceph-manager, Ceph-mon, Ceph-osd altogether three)

10.6.0.140 = Ceph-manager

10.6.0.187 = ceph-mon-1

10.6.0.188 = ceph-osd-1

10.6.0.94 = node-94

Note: Ceph is very time-critical, so be sure to synchronize all server time

First, modify the/etc/hosts on the manager:

10.6.0.187 ceph-mon-1
10.6.0.188 ceph-osd-1
10.6.0.94 node-94

Second, and modify the server above the hostname (NOTE: Ceph-deploy tools are the host name and other nodes to communicate)

Hostnamectl--static Set-hostname Ceph-manager
Hostnamectl--static Set-hostname ceph-mon-1
Hostnamectl--static Set-hostname ceph-osd-1
Hostnamectl--static Set-hostname node-94

Third, configure the manager node and other nodes SSH key access

[Email protected] ~]# Ssh-keygen

Send key to each node

[[Email protected] ~] #ssh-copy-id ceph-mon-1
[[Email protected] ~] #ssh-copy-id ceph-osd-1

Iv. installing Ceph-deploy on the manager node

[[Email protected] ~] #yum-y install Centos-release-ceph
[[Email protected] ~] #yum Makecache
[[Email protected] ~] #yum-y install Ceph-deploy ntpdate

Installing Ceph's Yum source on other nodes

[Email protected] ~]# yum-y install Centos-release-ceph
[email protected] ~]# Yum Makecache

[Email protected] ~]# yum-y install Centos-release-ceph
[email protected] ~]# Yum Makecache

V. Configuring a CEPH Cluster

Create a Ceph directory
[[Email protected] ~] #mkdir-P/etc/ceph
[[Email protected] ~] #cd/etc/ceph

To create a monitoring node:
[[email protected]/etc/ceph] #ceph-deploy new ceph-mon-1

Complete execution generates ceph.conf Ceph.log ceph.mon.keyring three files

Edit Ceph.conf Increase the number of OSD nodes
In the last addition:
OSD Pool Default size = 1

Install Ceph on all machines using Ceph-deploy

[Email protected]/etc/ceph]# ceph-deploy install Ceph-manager ceph-mon-1 ceph-osd-1

If an error occurs, you can also install it directly in each node by yum-y install Ceph CEPH-RADOSGW

Initializing the Monitoring node

[Email protected]/etc/ceph]# Ceph-deploy Mon create-initial

OSD Node Create storage space

[Email protected] ~]# mkdir-p/OPT/OSD1

Launch and activate the OSD process on the management node

[Email protected] ~]# Ceph-deploy OSD Prepare CEPH-OSD-1:/OPT/OSD1
[Email protected] ~]# Ceph-deploy OSD Activate CEPH-OSD-1:/OPT/OSD1

Synchronizing the configuration files of the management node with the keyring to other nodes

[Email protected] ~]# Ceph-deploy admin ceph-mon-1 ceph-osd-1

View Cluster health status (HEALTH_OK indicates OK)

[Email protected] ~]# Ceph Health
Health_ok

Client mount: Ceph has various mounting methods, RBD block device mapping, CEPHFS mount, etc.

Note:
In a production environment, the client should correspond to the pool's permissions, not the admin permissions

[Email protected] ~]# Ssh-copy-id node-94

Client Install Ceph
[Email protected] ~]# Ceph-deploy install node-94

or login node-94 execute yum-y install ceph CEPH-RADOSGW

If SSH is not port 22, an error can be sent using SCP

Scp-p Port ceph.conf node-94:/etc/ceph/
Scp-p Port ceph.client.admin.keyring node-94:/etc/ceph/

Create a pool
[[email protected] ~]# ceph OSD Pool Create press 100
Pool ' Press ' created

Set the pgp_num of the pool
[[email protected] ~]# ceph OSD Pool Set Press Pgp_num 100

View the pool created
[Email protected] ~]# ceph OSD Lspools
0 rbd,1 Press,

Set the number of copies to 2 (OSD must be greater than or equal to the number of copies, or error, please note)
[[email protected] ~]# ceph OSD Pool Set Press size 2

Create a 100G named image image
[Email protected] ~]# RBD create-p Press--size 100000 image

Take a look at the image:
[Email protected] ~]# Rbd-p Press info image
RBD image ' image ':
Size 100000 MB in 25000 objects
Order (4096 KB objects)
Block_name_prefix:rb.0.104b.74b0dc51
Format:1

Client Block Storage mount:
Map image on node-94
[Email protected] ~]# rbd-p Press map image
/dev/rbd0

Format image
[Email protected] ~]# mkfs.xfs/dev/rbd0

Create mount Directory
[Email protected] ~]# MKDIR/OPT/RBD

Mount RBD
[Email protected] ~]# MOUNT/DEV/RBD0/OPT/RBD

[[Email protected] ~]# time DD If=/dev/zero Of=haha bs=1m count=1000

Cancel map image
[Email protected] ~]# UMOUNT/OPT/RBD
[Email protected] ~]# RBD unmap/dev/rbd0

Client CEPHFS file System (CEPHFS must have 2 osd to run, please note):

MDS service must be available in a CEPHFS cluster

Create MDS Service (the MDS service is created on Mon's server due to machine limitations)
[Email protected] ~]# Ceph-deploy MDS Create ceph-mon-1

Create 2 pool as a file system data and metadata

[[email protected] ~]# ceph OSD Pool Create Cephfs_data 99
Pool ' Cephfs_data ' created

[[email protected] ~]# ceph OSD Pool Create Cephfs_metadata 99
Pool ' Cephfs_metadata ' created

To create a file system:
[[email protected] ~]# ceph FS new Jicki Cephfs_metadata Cephfs_data
New FS with metadata pool 6 and data Pool 5

View all file systems:
[[email protected] ~]# Ceph FS ls
Name:jicki, metadata pool:cephfs_metadata, data pools: [Cephfs_data]

Delete a file system
[[email protected] ~]# Ceph FS RM Jicki--yes-i-really-mean-it

Client Mount CEPHFS

Install Ceph-fuse:
[email protected] ~]# Yum install ceph-fuse-y

To create a mount directory:
[Email protected] ~]# mkdir-p/opt/jicki
[Email protected] ~]# Ceph-fuse/opt/jicki
[Email protected] ~]# Df-h|grep ceph
Ceph-fuse 1.6T 25G 1.6T 2%/opt/jicki

Based on Docker Ceph-rbd-volume-plugin

Vi. Ceph-related commands:

First, the manager chapter

To view real-time health status information:
[Email protected] ~]# ceph-w

To view status information:
[Email protected] ~]# ceph-s

To view storage space:
[[email protected] ~]# Ceph DF

Delete all the ceph packets for a node:
[Email protected] ~]# Ceph-deploy purge ceph-mon-1
[Email protected] ~]# Ceph-deploy purgedata ceph-mon-1

Create an Admin user for Ceph and create a key for the admin user to save the key to the/etc/ceph directory:
[[email protected] ~]# ceph auth get-or-create client.admin mds ' Allow ' OSD ' Allow * ' mon ' Allow * '-o/etc/ceph/ceph.clie Nt.admin.keyring

Create a user for osd.ceph-osd-1 and create a key
[[email protected] ~]# ceph auth get-or-create osd.ceph-osd-1 mon ' Allow rwx ' OSD ' Allow * '-o/etc/ceph/keyring

Create a user for mds.ceph-mon-1 and create a key
[[email protected] ~]# ceph auth get-or-create mds.ceph-mon-1 mon ' Allow rwx ' OSD ' Allow * ' MDS ' allow * '-o/etc/ceph/key Ring

View certified users and associated keys in a ceph cluster
[Email protected] ~]# Ceph Auth List

Remove an authenticated user from the cluster
[Email protected] ~]# ceph auth del osd.0

View Cluster health status details
[Email protected] ~]# ceph Health Detail

View the directory where the Ceph log log is located
[Email protected] ~]# ceph-conf--name mds.ceph-manager--show-config-value log_file

Second, Mon Chapter

View the status information for Mon
[Email protected] ~]# Ceph Mon stat

View Mon's election status
[Email protected] ~]# ceph quorum_status--format json-pretty

Look at the mapping information for Mon
[Email protected] ~]# ceph Mon dump

Delete a Mon node
[Email protected] ~]# ceph Mon Remove ceph-mon-1

Get a running Mon map and save it in the Mon-1-map.txt file
[Email protected] ~]# ceph Mon getmap-o mon-1-map.txt

View Mon-1-map.txt
[Email protected] ~]# Monmaptool--print mon-1-map.txt

Inject the Mon map above into the newly added node
[Email protected] ~]# ceph-mon-i ceph-mon-3--inject-monmap mon-1-map.txt

View Mon's socket
[Email protected] ~]# ceph-conf--name mon.ceph-mon-1--show-config-value admin_socket

View the detailed status of Mon
[Email protected] ~]# ceph daemon mon.ceph-mon-1 mon_status

Delete a Mon node
[Email protected] ~]# ceph Mon Remove ceph-mon-1

Third, MSD article

View MSD Status
[[email protected] ~]# ceph MDS Dump

Delete an MDS node
[[email protected] ~]# Ceph MDS RM 0 Mds.ceph-mds-1

Four, the OSD chapter

View the Ceph OSD running status
[[email protected] ~]# Ceph OSD Stat

viewing OSD mapping information
[[email protected] ~]# Ceph OSD Stat

View the directory tree of the OSD
[[email protected] ~]# Ceph OSD Tree

Down an OSD hard drive (Ceph OSD tree can view the OSD's hard disk information, below is the down osd.0 node)
[[email protected] ~]# Ceph OSD down 0

Remove an OSD hard disk from the cluster
[[email protected] ~]# Ceph OSD RM 0

Delete an OSD hard disk in the cluster and crush map to clear the map information
[Email protected] ~]# ceph OSD Crush RM osd.0

Delete the host node of an OSD in the cluster
[Email protected] ~]# ceph OSD Crush RM ceph-osd-1

View the maximum number of OSD
[Email protected] ~]# ceph OSD Getmaxosd

Sets the maximum number of OSD (this value must be enlarged when the OSD node is enlarged)
[[email protected] ~]# ceph OSD Setmaxosd 10

Set the weight of the OSD Crush Ceph OSD Crush set <ID> <WEIGHT> <NAME> ID WEIGHT NAME using ceph osd tree view
[[email protected] ~]# ceph OSD Crush set 1 3.0 host=ceph-osd-1

Set the weight of the OSD Ceph OSD Reweight <ID> <REWEIGHT>
[[email protected] ~]# ceph OSD Reweight 1 0.5

Kick an OSD node out of the cluster
[[email protected] ~]# Ceph OSD out Osd.1

Put the kicked out OSD back into the cluster
[[email protected] ~]# ceph OSD in Osd.1

Pause the OSD (the entire cluster will no longer receive data after pausing)
[[email protected] ~]# ceph OSD Pause

Turn on the OSD again (receive data again after opening)
[Email protected] ~]# ceph OSD Unpause

V. PG articles

View the mapping information for the PG group
[Email protected] ~]# Ceph PG Dump |more

View a map of PG
[Email protected] ~]# Ceph PG Map 0.3F

View PG Status
[Email protected] ~]# Ceph PG Stat

Query a PG for more information
[Email protected] ~]# Ceph PG 0.39 Query

View the status of the stuck in the PG (if an abnormal PG is displayed)
[Email protected] ~]# Ceph PG Dump_stuck Unclean
[Email protected] ~]# Ceph PG Dump_stuck Inactive
[Email protected] ~]# Ceph PG Dump_stuck Stale

Show all PG statistics in a cluster
[Email protected] ~]# Ceph PG Dump--format Plain|more

Recover a lost PG (Og-id for lost PG, using Ceph PG Dump_stuck Inactive|unclean|stale lookup)
[[email protected] ~]# Ceph PG {Pg-id} mark_unfound_lost revert

Six, the pool chapter

View the number of pool in a ceph cluster
[Email protected] ~]# ceph OSD Lspools

To view the PG group number:
[[email protected] ~]# ceph OSD Pool Get RBD pg_num

Create a pool in the Ceph cluster
[[email protected] ~]# ceph OSD Pool Create test 100 (name is test, 100 is PG Group number)

Configure quotas for a ceph pool
[Email protected] ~]# ceph OSD Pool Set-quota test max_objects 10000

Show all the pool
[Email protected] ~]# ceph OSD Pool ls

Remove a pool from the cluster
[[email protected] ~]# ceph OSD Pool Delete test test--yes-i-really-really-mean-it

Show the details of the pool in the cluster
[[email protected] ~]# Rados DF

Create a snapshot for a pool
[Email protected] ~]# ceph OSD Pool Mksnap test Test-snap

Delete a snapshot of a pool
[Email protected] ~]# ceph OSD Pool Rmsnap test Test-snap

View the number of PG in the data pool
[[email protected] ~]# ceph OSD Pool get test Pg_num

Set the maximum storage space for the data pool (default is 1T, 1T = 1000000000000, 100T below)
[[email protected] ~]# ceph OSD Pool Set test target_max_bytes 100000000000000

Set the number of copies of the data pool
[[email protected] ~]# ceph OSD Pool Set Test size 3

Set the minimum copy of the data pool to accept write operations to 2
[[email protected] ~]# ceph OSD Pool Set test Min_size 2

View the copy size of all pool in a cluster
[Email protected] ~]# ceph OSD Dump | grep ' Replicated size '

Set the number of PG for a pool
[[email protected] ~]# ceph OSD Pool set test Pg_num 100

Set the number of PGP for a pool
[[email protected] ~]# ceph OSD Pool set test Pgp_num 100

View Ceph object in the Ceph pool (volumes is the pool name) (here the object is stored as a block)
[Email protected] ~]# Rados ls-p Volumes | More

Create an Object
[Email protected] ~]# Rados create test-object-p test

View Object
[Email protected] ~]# rados-p test ls

Delete an Object
[Email protected] ~]# Rados RM test-object-p Test

View all mirrors in a pool in ceph (volumes is the pool name)
[Email protected] ~]# RBD LS volumes

Create a 1000M image named images in the test pool
[Email protected] ~]# RBD create-p test--size Images
View the image information you just created
[Email protected] ~]# RBD-P Test Info Images

Delete a mirror
[Email protected] ~]# RBD rm-p test Images

Adjust the size of a mirror
[Email protected] ~]# RBD resize-p test--size Images

Create a snapshot of the image (pool/Mirror name @ snapshot name)
[[email protected] ~]# RBD snap create Test/[email protected]

Delete a snapshot of a mirrored file
[[email protected] ~]# RBD snap rm snapshot Pool/Snapshot image file @ specific snapshot

If you delete the snapshot prompt protection, you need to remove the protection first
[[email protected] ~]# RBD snap unprotect snapshot pool/snapshot image file @ specific snapshot

Delete all snapshots of an image file
[[email protected] ~]# RBD snap purge-p snapshot pool/snapshot image file

Export a mirror from the Ceph pool
[Email protected] ~]# RBD export-p images--image < specific image id>/tmp/images.img

Ceph RBD CEPHFS Storage