一、Ceph简介
1. Ceph的几种存储类型
Ceph是一种开源的分布式的存储系统，包含以下几种存储类型：
块存储（rbd），对象存储(RADOS Fateway)，文件系统（Cephfs）
1）. 块存储（rbd）
块是一个字节序列（例如，512字节的数据块）。 基于块的存储接口是使用旋转介质（如硬盘，CD，软盘甚至传统的9轨磁带）是存储数据的最常用方法。
Ceph块设备是精简配置，可调整大小并存储在Ceph集群中多个OSD条带化的数据。 Ceph块设备利用RADOS功能，如快照，复制和一致性。 Ceph的RADOS块设备（RBD）使用内核模块或librbd库与OSD进行交互。可以使用同一个集群同时运行Ceph RADOS Gateway，CephFS文件系统和Ceph块设备。
Linux系统中，ls /dev/下有很多块设备文件，这些文件就是添加硬盘时识别出来的；rbd就是由Ceph集群提供出来的块设备。即：sda是通过数据线连接到了真实的硬盘，而rbd是通过网络连接到了Ceph集群中的一块存储区域，往rbd设备文件写入数据，最终会被存储到Ceph集群的这块区域中。
总结：块设备可理解成一块硬盘，用户可以直接使用不含文件系统的块设备，也可以将其格式化成特定的文件系统，由文件系统来组织管理存储空间，从而为用户提供丰富而友好的数据操作支持。
2）. 文件系统CephFS
Ceph文件系统（CephFS）是一个符合POSIX标准的文件系统，它使用Ceph存储集群来存储其数据。 Ceph文件系统使用与Ceph块设备相同的Ceph存储集群系统。用户可以在块设备上创建xfs文件系统，也可以创建ext4等其他文件系统，Ceph集群实现了自己的文件系统来组织管理集群的存储空间，用户可以直接将Ceph集群的文件系统挂载到用户机上使用。
Ceph有了块设备接口，在块设备上完全可以构建一个文件系统，那么Ceph为什么还需要文件系统接口呢？主要是因为应用场景的不同，Ceph的块设备具有优异的读写性能，但不能多处挂载同时读写，目前主要用在OpenStack上作为虚拟磁盘，而Ceph的文件系统接口读写性能较块设备接口差，但具有优异的共享性。
3）. 对象存储
Ceph对象存储使用Ceph对象网关守护进程（radosgw），它是一个用于与Ceph存储集群交互的HTTP服务器。由于它提供与OpenStack Swift和Amazon S3兼容的接口，因此Ceph对象网关具有自己的用户管理。 Ceph对象网关可以将数据存储在用于存储来自Ceph文件系统客户端或Ceph块设备客户端的数据的相同Ceph存储集群中，使用方式就是通过http协议上传下载删除对象（文件即对象）。
那么，有了块设备接口存储和文件系统接口存储，为什么还整个对象存储呢? 是因为Ceph的块设备存储具有优异的存储性能但不具有共享性，而Ceph的文件系统具有共享性然而性能较块设备存储差，而对象存储既具有共享性且存储性能又好于文件系统存储。
2. 分布式存储的优点
高可靠：既满足存储读取不丢失，还要保证数据长期存储。 在保证部分硬件损坏后依然可以保证数据安全
高性能：读写速度快
可扩展：分布式存储的优势就是“分布式”，所谓的“分布式”就是能够将多个物理节点整合在一起形成共享的存储池，节点可以线性扩充，这样可以源源不断的通过扩充节点提升性能和扩大容量。
二、Ceph核心组件
Ceph存储集群至少需要一个Ceph Monitor，Ceph Manager和Ceph OSD（对象存储守护进程）。 运行Ceph Filesystem客户端时也需要Ceph元数据服务器。
1. Ceph Monitor
Ceph Monitor 维护整个ceph的状态，是Ceph的监视器（ceph-mon）维护集群状态的映射，包括监视器映射，管理器映射，OSD映射和CRUSH映射。这些映射是Ceph守护进程相互协调所需的关键集群状态。监视器还负责管理守护进程和客户端之间的身份验证。冗余和高可用性通常至少需要三个监视器。
2. Ceph Manager
Ceph Manager守护程序（ceph-mgr）负责跟踪运行时指标和Ceph集群的当前状态，包括存储利用率，当前性能指标和系统负载。 Ceph Manager守护进程还托管基于python的模块来管理和公开Ceph集群信息，包括基于Web的Ceph Dashboard和REST API。ceph集群的运行指标统计，可以通过web显示ceph集群的运行状态，包括存储池信息、集群IO和负载。高可用性通常至少需要两名Managers。
3. Ceph OSD
Ceph OSD（对象存储守护进程，ceph-osd）存储数据，可以理解为一块磁盘创建一个OSDs，每个磁盘都会启动一个OSD进程。处理数据复制，恢复，重新平衡，并通过检查其他Ceph OSD守护进程来获取心跳，为Ceph监视器和管理器提供一些监视信息。冗余和高可用性通常至少需要3个Ceph OSD。
4. MDS
Ceph元数据服务器（MDS，ceph-mds）代表Ceph文件系统存储元数据（即，Ceph块设备和Ceph对象存储不使用MDS，它们的元数据是直接写到磁盘上的）。 Ceph元数据服务器允许POSIX文件系统用户执行基本命令（如ls，find等），而不会给Ceph存储集群带来巨大负担。
三、安装Ceph集群
1. 环境准备
机器配置：CentOS 7.9；网络模式：NAT；准备三台机器，每台机器需要两块硬盘
2. 初始化实验环境
1）配置hosts文件
在上述三台机器上分别执行如下命令，修改/etc/hosts文件内容：
cat >>/etc/hosts <<EOF
> 10.0.0.130 ceph-admin
> 10.0.0.129 ceph-monitor
> 10.0.0.128 ceph-osd
>
> EOF
cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.130 ceph-admin
10.0.0.129 ceph-monitor
10.0.0.128 ceph-osd2）配置互信
生成ssh 密钥对，并将本地的ssh公钥文件上传至其他主机
[root@ceph-admin ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Pwig0NCfayeXYJDPt3SSco9txTs2KpeHk3U5I1CCggY root@ceph-admin
The key's randomart image is:
+---[RSA 2048]----+
|Eo o . |
| o* . . . . |
|...=.o . + |
| . .O.* o o |
| .. B.OSo . . |
| + *.+oB = |
| . + ..Bo= o |
| . B .. |
| o o |
+----[SHA256]-----+
#上传ssh公钥文件
[root@ceph-admin ~]# ssh-copy-id ceph-admin
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'ceph-admin (10.0.0.130)' can't be established.
ECDSA key fingerprint is SHA256:J9UnR8HG9Iws8xvmhv4HMjfjJUgOGgEV/3yQ/kFT87c.
ECDSA key fingerprint is MD5:af:38:29:b9:6b:1c:eb:03:bd:93:ad:0d:5a:68:4d:06.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@ceph-admin's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'ceph-admin'"
and check to make sure that only the key(s) you wanted were added.
[root@ceph-admin ~]# ssh-copy-id ceph-monitor
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'ceph-monitor (10.0.0.129)' can't be established.
ECDSA key fingerprint is SHA256:J9UnR8HG9Iws8xvmhv4HMjfjJUgOGgEV/3yQ/kFT87c.
ECDSA key fingerprint is MD5:af:38:29:b9:6b:1c:eb:03:bd:93:ad:0d:5a:68:4d:06.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@ceph-monitor's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'ceph-monitor'"
and check to make sure that only the key(s) you wanted were added.
[root@ceph-admin ~]# ssh-copy-id ceph-osd
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'ceph-osd (10.0.0.128)' can't be established.
ECDSA key fingerprint is SHA256:J9UnR8HG9Iws8xvmhv4HMjfjJUgOGgEV/3yQ/kFT87c.
ECDSA key fingerprint is MD5:af:38:29:b9:6b:1c:eb:03:bd:93:ad:0d:5a:68:4d:06.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@ceph-osd's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'ceph-osd'"
and check to make sure that only the key(s) you wanted were added.
[root@ceph-admin ~]#其他节点同理操作。
3）配置Ceph安装源
[root@ceph-admin ~]# cat /etc/yum.repos.d/ceph.repo
[Ceph]
name=Ceph packages for $basearch
baseurl=http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/x86_64/
enabled=1
gpgcheck=0
type=rpm-md
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/noarch/
enabled=1
gpgcheck=0
type=rpm-md
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/SRPMS/
enabled=1
gpgcheck=0
type=rpm-md
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc
priority=1检测源的可用性
[root@ceph-admin ~]# yum makecache
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink | 9.4 kB 00:00:00
* epel: mirror.lzu.edu.cn
Ceph | 2.9 kB 00:00:00
Ceph-noarch | 2.9 kB 00:00:00
base | 3.6 kB 00:00:00
ceph-source | 2.9 kB 00:00:00
docker-ce-stable | 3.5 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/9): Ceph/filelists_db | 269 kB 00:00:00
(2/9): Ceph/primary_db | 329 kB 00:00:00
(3/9): Ceph/other_db | 28 kB 00:00:00
(4/9): Ceph-noarch/filelists_db | 13 kB 00:00:00
(5/9): Ceph-noarch/other_db | 1.9 kB 00:00:00
(6/9): ceph-source/primary_db | 20 kB 00:00:00
(7/9): ceph-source/other_db | 2.8 kB 00:00:00
(8/9): ceph-source/filelists_db | 3.3 kB 00:00:00
(9/9): Ceph-noarch/primary_db | 6.4 kB 00:00:01
Metadata Cache Created
#查看源列表
[root@ceph-admin ~]# yum repolist -v
Loading "fastestmirror" plugin
Config time: 0.009
Yum version: 3.4.3
Loading mirror speeds from cached hostfile
* epel: mirror.lzu.edu.cn
Setting up Package Sacks
pkgsack time: 0.101
Repo-id : Ceph
Repo-name : Ceph packages for x86_64
Repo-revision: 1531250891
Repo-updated : Wed Jul 11 03:32:26 2018
Repo-pkgs : 499
Repo-size : 20 G
Repo-baseurl : http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/x86_64/
Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:53 2022)
Filter : read-only:present
Repo-filename: /etc/yum.repos.d/ceph.repo
Repo-id : Ceph-noarch
Repo-name : Ceph noarch packages
Repo-revision: 1531250880
Repo-updated : Wed Jul 11 03:28:04 2018
Repo-pkgs : 16
Repo-size : 3.2 M
Repo-baseurl : http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/noarch/
Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:53 2022)
Filter : read-only:present
Repo-filename: /etc/yum.repos.d/ceph.repo
Repo-id : base/7/x86_64
Repo-name : CentOS-7 - Base
Repo-revision: 1604001756
Repo-updated : Fri Oct 30 04:03:00 2020
Repo-pkgs : 10,072
Repo-size : 8.9 G
Repo-baseurl : https://mirrors.tuna.tsinghua.edu.cn/centos/7/os/x86_64/
Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022)
Filter : read-only:present
Repo-filename: /etc/yum.repos.d/CentOS-Base.repo
Repo-id : ceph-source
Repo-name : Ceph source packages
Repo-revision: 1531250885
Repo-updated : Wed Jul 11 03:28:11 2018
Repo-pkgs : 0
Repo-size : 0
Repo-baseurl : http://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-jewel/el7/SRPMS/
Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022)
Filter : read-only:present
Repo-filename: /etc/yum.repos.d/ceph.repo
Repo-id : docker-ce-stable/7/x86_64
Repo-name : Docker CE Stable - x86_64
Repo-revision: 1662713508
Repo-updated : Fri Sep 9 16:51:48 2022
Repo-pkgs : 169
Repo-size : 3.8 G
Repo-baseurl : https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/7/x86_64/stable/
Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022)
Filter : read-only:present
Repo-filename: /etc/yum.repos.d/docker-ce.repo
Repo-id : epel/x86_64
Repo-name : Extra Packages for Enterprise Linux 7 - x86_64
Repo-revision: 1664157257
Repo-updated : Mon Sep 26 10:14:48 2022
Repo-pkgs : 13,752
Repo-size : 16 G
Repo-metalink: https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos
Updated : Mon Sep 26 10:14:48 2022
Repo-baseurl : https://mirror.lzu.edu.cn/epel/7/x86_64/ (22 more)
Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022)
Filter : read-only:present
Repo-filename: /etc/yum.repos.d/epel.repo
Repo-id : extras/7/x86_64
Repo-name : CentOS-7 - Extras
Repo-revision: 1662061316
Repo-updated : Fri Sep 2 03:41:58 2022
Repo-pkgs : 516
Repo-size : 1.0 G
Repo-baseurl : https://mirrors.tuna.tsinghua.edu.cn/centos/7/extras/x86_64/
Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022)
Filter : read-only:present
Repo-filename: /etc/yum.repos.d/CentOS-Base.repo
Repo-id : updates/7/x86_64
Repo-name : CentOS-7 - Updates
Repo-revision: 1663935737
Repo-updated : Fri Sep 23 20:23:22 2022
Repo-pkgs : 4,244
Repo-size : 28 G
Repo-baseurl : https://mirrors.tuna.tsinghua.edu.cn/centos/7/updates/x86_64/
Repo-expire : 21,600 second(s) (last: Tue Sep 27 22:36:54 2022)
Filter : read-only:present
Repo-filename: /etc/yum.repos.d/CentOS-Base.repo
repolist: 29,268
[root@ceph-admin ~]#将配置好的ceph.repo源文件上传至ceph-monitor和ceph-osd节点上
[root@ceph-admin ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.129:/etc/yum.repos.d/ceph.repo
ceph.repo 100% 681 665.8KB/s 00:00
[root@ceph-admin ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.128:/etc/yum.repos.d/ceph.repo
ceph.repo 100% 681 256.6KB/s 00:00
[root@ceph-admin ~]#3. 安装ceph集群
1）安装插件ceph-deploy,ceph,ceph-radosgw
在ceph-admin节点安装ceph-deploy
[root@ceph-admin ~]# yum install python-setuptools ceph-deploy -y在ceph-admin、ceph-monitor和ceph-osd节点安装ceph
[root@ceph-admin ~]# yum install ceph ceph-radosgw -y
[root@ceph-osd ~]# yum install ceph ceph-radosgw -y
[root@ceph-monitor ~]# yum install ceph ceph-radosgw -y查看ceph版本
[root@ceph-admin ~]# ceph --version
ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)
[root@ceph-admin ~]#2）创建monitor节点
a. 创建一个目录，用于保存 ceph-deploy 生成的配置文件信息的
[root@ceph-admin ~]# cd /etc/ceph
[root@ceph-admin ceph]# ll
total 4
-rw-r--r-- 1 root root 92 Jul 10 2018 rbdmap
[root@ceph-admin ceph]# ceph-deploy new ceph-admin ceph-monitor ceph-osd
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy new ceph-admin ceph-monitor ceph-osd
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] func : <function new at 0x7f91d7f70758>
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f91d76dcf38>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] ssh_copykey : True
[ceph_deploy.cli][INFO ] mon : ['ceph-admin', 'ceph-monitor', 'ceph-osd']
[ceph_deploy.cli][INFO ] public_network : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] cluster_network : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] fsid : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph-admin][DEBUG ] connected to host: ceph-admin
[ceph-admin][DEBUG ] detect platform information from remote host
[ceph-admin][DEBUG ] detect machine type
[ceph-admin][DEBUG ] find the location of an executable
[ceph-admin][INFO ] Running command: /usr/sbin/ip link show
[ceph-admin][INFO ] Running command: /usr/sbin/ip addr show
[ceph-admin][DEBUG ] IP addresses found: [u'172.17.0.1', u'10.0.0.130']
[ceph_deploy.new][DEBUG ] Resolving host ceph-admin
[ceph_deploy.new][DEBUG ] Monitor ceph-admin at 10.0.0.130
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph-monitor][DEBUG ] connected to host: ceph-admin
[ceph-monitor][INFO ] Running command: ssh -CT -o BatchMode=yes ceph-monitor
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] find the location of an executable
[ceph-monitor][INFO ] Running command: /usr/sbin/ip link show
[ceph-monitor][INFO ] Running command: /usr/sbin/ip addr show
[ceph-monitor][DEBUG ] IP addresses found: [u'10.0.0.129', u'172.17.0.1']
[ceph_deploy.new][DEBUG ] Resolving host ceph-monitor
[ceph_deploy.new][DEBUG ] Monitor ceph-monitor at 10.0.0.129
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph-osd][DEBUG ] connected to host: ceph-admin
[ceph-osd][INFO ] Running command: ssh -CT -o BatchMode=yes ceph-osd
[ceph-osd][DEBUG ] connected to host: ceph-osd
[ceph-osd][DEBUG ] detect platform information from remote host
[ceph-osd][DEBUG ] detect machine type
[ceph-osd][DEBUG ] find the location of an executable
[ceph-osd][INFO ] Running command: /usr/sbin/ip link show
[ceph-osd][INFO ] Running command: /usr/sbin/ip addr show
[ceph-osd][DEBUG ] IP addresses found: [u'10.0.0.128', u'172.17.0.1']
[ceph_deploy.new][DEBUG ] Resolving host ceph-osd
[ceph_deploy.new][DEBUG ] Monitor ceph-osd at 10.0.0.128
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-admin', 'ceph-monitor', 'ceph-osd']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.0.0.130', '10.0.0.129', '10.0.0.128']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...查看共生成Ceph配置文件、一个monitor密钥环和一个日志文件
[root@ceph-admin ceph]# ls -lrt
total 20
-rw-r--r-- 1 root root 92 Jul 10 2018 rbdmap
-rw------- 1 root root 73 Sep 27 22:59 ceph.mon.keyring
-rw-r--r-- 1 root root 5115 Sep 27 22:59 ceph-deploy-ceph.log
-rw-r--r-- 1 root root 244 Sep 27 22:59 ceph.conf
[root@ceph-admin ceph]#b. 修改ceph配置文件，修改以下内容
#将ceph.conf配置文件里的默认副本数从3改成1
#将osd_pool_default_size = 2 加入[global]段，这样只有2个osd也能达到active+clean状态
#mon clock drift allowed #监视器间允许的时钟漂移量默认值0.05
#mon clock drift warn backoff #时钟偏移警告的退避指数。默认值5
[root@ceph-admin ceph]# vim ceph.conf
You have new mail in /var/spool/mail/root
[root@ceph-admin ceph]# cat ceph.conf
[global]
fsid = 75bf70df-8562-4f70-bb28-db9376a1ce98
mon_initial_members = ceph-monitor
mon_host = 10.0.0.129
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd_pool_default_size = 2
mon clock drift allowed = 0.500
mon clock drift warn backoff = 10
[root@ceph-admin ceph]#ceph对每个monitor之间的时间同步延时默认要求在0.05s之间，这个时间有的时候太短了。所以如果ceph集群如果出现clock问题就检查ntp时间同步或者适当放宽这个误差时间。
cephx是认证机制是整个 Ceph 系统的用户名/密码
c. 配置初始monitor、收集所有的密钥
[root@ceph-admin ceph]# ceph-deploy mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create-initial
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f69f5c49488>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mon at 0x7f69f5c2e848>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] keyrings : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-monitor
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-monitor ...
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO ] distro info: CentOS Linux 7.9.2009 Core
[ceph-monitor][DEBUG ] determining if provided host has same hostname in remote
[ceph-monitor][DEBUG ] get remote short hostname
[ceph-monitor][DEBUG ] deploying mon to ceph-monitor
[ceph-monitor][DEBUG ] get remote short hostname
[ceph-monitor][DEBUG ] remote hostname: ceph-monitor
[ceph-monitor][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-monitor][DEBUG ] create the mon path if it does not exist
[ceph-monitor][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-monitor/done
[ceph-monitor][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-monitor/done
[ceph-monitor][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-monitor.mon.keyring
[ceph-monitor][DEBUG ] create the monitor keyring file
[ceph-monitor][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i ceph-monitor --keyring /var/lib/ceph/tmp/ceph-ceph-monitor.mon.keyring --setuser 167 --setgroup 167
[ceph-monitor][DEBUG ] ceph-mon: mon.noname-a 10.0.0.129:6789/0 is local, renaming to mon.ceph-monitor
[ceph-monitor][DEBUG ] ceph-mon: set fsid to 75bf70df-8562-4f70-bb28-db9376a1ce98
[ceph-monitor][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-monitor for mon.ceph-monitor
[ceph-monitor][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-monitor.mon.keyring
[ceph-monitor][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-monitor][DEBUG ] create the init path if it does not exist
[ceph-monitor][INFO ] Running command: systemctl enable ceph.target
[ceph-monitor][INFO ] Running command: systemctl enable ceph-mon@ceph-monitor
[ceph-monitor][WARNIN] Created symlink from /etc/systemd/system/ceph-mon.target.wants/ceph-mon@ceph-monitor.service to /usr/lib/systemd/system/ceph-mon@.service.
[ceph-monitor][INFO ] Running command: systemctl start ceph-mon@ceph-monitor
[ceph-monitor][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-monitor.asok mon_status
[ceph-monitor][DEBUG ] ********************************************************************************
[ceph-monitor][DEBUG ] status for monitor: mon.ceph-monitor
[ceph-monitor][DEBUG ] {
[ceph-monitor][DEBUG ] "election_epoch": 3,
[ceph-monitor][DEBUG ] "extra_probe_peers": [],
[ceph-monitor][DEBUG ] "monmap": {
[ceph-monitor][DEBUG ] "created": "2022-09-27 23:06:45.440351",
[ceph-monitor][DEBUG ] "epoch": 1,
[ceph-monitor][DEBUG ] "fsid": "75bf70df-8562-4f70-bb28-db9376a1ce98",
[ceph-monitor][DEBUG ] "modified": "2022-09-27 23:06:45.440351",
[ceph-monitor][DEBUG ] "mons": [
[ceph-monitor][DEBUG ] {
[ceph-monitor][DEBUG ] "addr": "10.0.0.129:6789/0",
[ceph-monitor][DEBUG ] "name": "ceph-monitor",
[ceph-monitor][DEBUG ] "rank": 0
[ceph-monitor][DEBUG ] }
[ceph-monitor][DEBUG ] ]
[ceph-monitor][DEBUG ] },
[ceph-monitor][DEBUG ] "name": "ceph-monitor",
[ceph-monitor][DEBUG ] "outside_quorum": [],
[ceph-monitor][DEBUG ] "quorum": [
[ceph-monitor][DEBUG ] 0
[ceph-monitor][DEBUG ] ],
[ceph-monitor][DEBUG ] "rank": 0,
[ceph-monitor][DEBUG ] "state": "leader",
[ceph-monitor][DEBUG ] "sync_provider": []
[ceph-monitor][DEBUG ] }
[ceph-monitor][DEBUG ] ********************************************************************************
[ceph-monitor][INFO ] monitor: mon.ceph-monitor is running
[ceph-monitor][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-monitor.asok mon_status
[ceph_deploy.mon][INFO ] processing monitor mon.ceph-monitor
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] find the location of an executable
[ceph-monitor][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-monitor.asok mon_status
[ceph_deploy.mon][INFO ] mon.ceph-monitor monitor has reached quorum!
[ceph_deploy.mon][INFO ] all initial monitors are running and have formed quorum
[ceph_deploy.mon][INFO ] Running gatherkeys...
[ceph_deploy.gatherkeys][INFO ] Storing keys in temp directory /tmp/tmphUcJ_k
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] get remote short hostname
[ceph-monitor][DEBUG ] fetch remote file
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.ceph-monitor.asok mon_status
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.admin
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.bootstrap-mds
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.bootstrap-mgr
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get-or-create client.bootstrap-mgr mon allow profile bootstrap-mgr
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.bootstrap-osd
[ceph-monitor][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-monitor/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mgr.keyring
[ceph_deploy.gatherkeys][INFO ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmphUcJ_k执行上述命令后，生成以下密钥文件
[root@ceph-admin ceph]# ls -lrt
total 48
-rw-r--r-- 1 root root 92 Jul 10 2018 rbdmap
-rw------- 1 root root 73 Sep 27 22:59 ceph.mon.keyring
-rw-r--r-- 1 root root 292 Sep 27 23:04 ceph.conf
-rw------- 1 root root 129 Sep 27 23:06 ceph.client.admin.keyring
-rw------- 1 root root 113 Sep 27 23:06 ceph.bootstrap-mds.keyring
-rw------- 1 root root 71 Sep 27 23:06 ceph.bootstrap-mgr.keyring
-rw------- 1 root root 113 Sep 27 23:06 ceph.bootstrap-osd.keyring
-rw------- 1 root root 113 Sep 27 23:06 ceph.bootstrap-rgw.keyring
-rw-r--r-- 1 root root 15385 Sep 27 23:06 ceph-deploy-ceph.log
[root@ceph-admin ceph]#3）部署osd服务
#准备osd
[root@ceph-admin ceph]# ceph-deploy osd prepare ceph-admin:/dev/sdb
[root@ceph-admin ceph]# ceph-deploy osd prepare ceph-monitor:/dev/sdb
[root@ceph-admin ceph]# ceph-deploy osd prepare ceph-osd:/dev/sdb#激活osd
[root@ceph-admin ceph]# ceph-deploy osd activate ceph-admin:/dev/sdb
[root@ceph-admin ceph]# ceph-deploy osd activate ceph-monitor:/dev/sdb
[root@ceph-admin ceph]# ceph-deploy osd activate ceph-osd:/dev/sdb#查看状态
[root@ceph-admin ceph]# ceph-deploy osd list ceph-admin ceph-monitor ceph-osd
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy osd list ceph-admin ceph-monitor ceph-osd
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : list
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f6818351680>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function osd at 0x7f68183a3140>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : [('ceph-admin', None, None), ('ceph-monitor', None, None), ('ceph-osd', None, None)]
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] find the location of an executable
[ceph-monitor][DEBUG ] find the location of an executable
[ceph-monitor][INFO ] Running command: /bin/ceph --cluster=ceph osd tree --format=json
[ceph-admin][DEBUG ] connected to host: ceph-admin
[ceph-admin][DEBUG ] detect platform information from remote host
[ceph-admin][DEBUG ] detect machine type
[ceph-admin][DEBUG ] find the location of an executable
[ceph-admin][INFO ] Running command: /usr/sbin/ceph-disk list
[ceph-admin][INFO ] ----------------------------------------
[ceph-admin][INFO ] ceph-0
[ceph-admin][INFO ] ----------------------------------------
[ceph-admin][INFO ] Path /var/lib/ceph/osd/ceph-0
[ceph-admin][INFO ] ID 0
[ceph-admin][INFO ] Name osd.0
[ceph-admin][INFO ] Status up
[ceph-admin][INFO ] Reweight 1.0
[ceph-admin][INFO ] Magic ceph osd volume v026
[ceph-admin][INFO ] Journal_uuid 9b5756c6-2d0d-4ae6-8b55-cb965ed3d69e
[ceph-admin][INFO ] Active ok
[ceph-admin][INFO ] Device /dev/sdb1
[ceph-admin][INFO ] Whoami 0
[ceph-admin][INFO ] Journal path /dev/sdb2
[ceph-admin][INFO ] ----------------------------------------
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph-monitor][DEBUG ] find the location of an executable
[ceph-monitor][INFO ] Running command: /usr/sbin/ceph-disk list
[ceph-monitor][INFO ] ----------------------------------------
[ceph-monitor][INFO ] ceph-1
[ceph-monitor][INFO ] ----------------------------------------
[ceph-monitor][INFO ] Path /var/lib/ceph/osd/ceph-1
[ceph-monitor][INFO ] ID 1
[ceph-monitor][INFO ] Name osd.1
[ceph-monitor][INFO ] Status up
[ceph-monitor][INFO ] Reweight 1.0
[ceph-monitor][INFO ] Magic ceph osd volume v026
[ceph-monitor][INFO ] Journal_uuid 655bc91d-f7d1-4145-b2c3-5d5f63c1a798
[ceph-monitor][INFO ] Active ok
[ceph-monitor][INFO ] Device /dev/sdb1
[ceph-monitor][INFO ] Whoami 1
[ceph-monitor][INFO ] Journal path /dev/sdb2
[ceph-monitor][INFO ] ----------------------------------------
[ceph-osd][DEBUG ] connected to host: ceph-osd
[ceph-osd][DEBUG ] detect platform information from remote host
[ceph-osd][DEBUG ] detect machine type
[ceph-osd][DEBUG ] find the location of an executable
[ceph-osd][INFO ] Running command: /usr/sbin/ceph-disk list
[ceph-osd][INFO ] ----------------------------------------
[ceph-osd][INFO ] ceph-2
[ceph-osd][INFO ] ----------------------------------------
[ceph-osd][INFO ] Path /var/lib/ceph/osd/ceph-2
[ceph-osd][INFO ] ID 2
[ceph-osd][INFO ] Name osd.2
[ceph-osd][INFO ] Status up
[ceph-osd][INFO ] Reweight 1.0
[ceph-osd][INFO ] Magic ceph osd volume v026
[ceph-osd][INFO ] Journal_uuid 7c5e1478-92ab-44d2-9445-620b0e535c71
[ceph-osd][INFO ] Active ok
[ceph-osd][INFO ] Device /dev/sdb1
[ceph-osd][INFO ] Whoami 2
[ceph-osd][INFO ] Journal path /dev/sdb2
[ceph-osd][INFO ] ----------------------------------------
[root@ceph-admin ceph]#4）创建ceph文件系统
要使用Ceph文件系统，Ceph的存储集群里至少需要存在一个Ceph的元数据服务器(mds)
#创建mds
[root@ceph-admin ceph]# ceph-deploy mds create ceph-admin ceph-monitor ceph-osd
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy mds create ceph-admin ceph-monitor ceph-osd
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f88afbd68c0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mds at 0x7f88afbad848>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] mds : [('ceph-admin', 'ceph-admin'), ('ceph-monitor', 'ceph-monitor'), ('ceph-osd', 'ceph-osd')]
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts ceph-admin:ceph-admin ceph-monitor:ceph-monitor ceph-osd:ceph-osd
[ceph-admin][DEBUG ] connected to host: ceph-admin
[ceph-admin][DEBUG ] detect platform information from remote host
[ceph-admin][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-admin
[ceph-admin][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-admin][WARNIN] mds keyring does not exist yet, creating one
[ceph-admin][DEBUG ] create a keyring file
[ceph-admin][DEBUG ] create path if it doesn't exist
[ceph-admin][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-admin osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-admin/keyring
[ceph-admin][INFO ] Running command: systemctl enable ceph-mds@ceph-admin
[ceph-admin][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-admin.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-admin][INFO ] Running command: systemctl start ceph-mds@ceph-admin
[ceph-admin][INFO ] Running command: systemctl enable ceph.target
[ceph-monitor][DEBUG ] connected to host: ceph-monitor
[ceph-monitor][DEBUG ] detect platform information from remote host
[ceph-monitor][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-monitor
[ceph-monitor][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-monitor][DEBUG ] create path if it doesn't exist
[ceph-monitor][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-monitor osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-monitor/keyring
[ceph-monitor][INFO ] Running command: systemctl enable ceph-mds@ceph-monitor
[ceph-monitor][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-monitor.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-monitor][INFO ] Running command: systemctl start ceph-mds@ceph-monitor
[ceph-monitor][INFO ] Running command: systemctl enable ceph.target
[ceph-osd][DEBUG ] connected to host: ceph-osd
[ceph-osd][DEBUG ] detect platform information from remote host
[ceph-osd][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-osd
[ceph-osd][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-osd][WARNIN] mds keyring does not exist yet, creating one
[ceph-osd][DEBUG ] create a keyring file
[ceph-osd][DEBUG ] create path if it doesn't exist
[ceph-osd][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-osd osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-osd/keyring
[ceph-osd][INFO ] Running command: systemctl enable ceph-mds@ceph-osd
[ceph-osd][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-osd.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-osd][INFO ] Running command: systemctl start ceph-mds@ceph-osd
[ceph-osd][INFO ] Running command: systemctl enable ceph.target#查看ceph当前文件系统
[root@ceph-admin ceph]# ceph fs ls
No filesystems enabled
[root@ceph-admin ceph]#一个cephfs至少要求两个librados存储池，一个为data，一个为metadata。当配置这两个存储池时，注意：
（1）为metadata pool设置较高级别的副本级别，因为metadata的损坏可能导致整个文件系统不用
（2）建议，metadata pool使用低延时存储，比如SSD，因为metadata会直接影响客户端的响应速度
#创建存储池
[root@ceph-admin ceph]# ceph osd pool create cephfs_data 128
pool 'cephfs_data' created
[root@ceph-admin ceph]# ceph osd pool create cephfs_metadata 128
pool 'cephfs_metadata' created
[root@ceph-admin ceph]#关于创建存储池
确定 pg_num 取值是强制性的，因为不能自动计算。下面是几个常用的值：
*少于 5 个 OSD 时可把 pg_num 设置为 128
*OSD 数量在 5 到 10 个时，可把 pg_num 设置为 512
*OSD 数量在 10 到 50 个时，可把 pg_num 设置为 4096
*OSD 数量大于 50 时，得理解权衡方法、以及如何计算 pg_num 取值,自己计算 pg_num 取值时可借助 pgcalc 工具
随着 OSD 数量的增加，正确的 pg_num 取值变得更加重要，因为它显著地影响着集群的行为、以及出错时的数据持久性（即灾难性事件导致数据丢失的概率）。
#创建文件系统
[root@ceph-admin ceph]# ceph fs new test-cephfs cephfs_metadata cephfs_data
new fs with metadata pool 2 and data pool 1其中：new后的fsname 可自定义
cephfs
[root@ceph-admin ceph]# ceph fs ls
name: test-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]#查看mds节点状态
[root@ceph-admin ceph]# ceph mds stat
e7: 1/1/1 up {0=ceph-osd=up:active}, 2 up:standby其中，active是活跃的，另1个是处于热备份的状态
#查看集群状态
[root@ceph-admin ceph]# ceph -s
cluster 75bf70df-8562-4f70-bb28-db9376a1ce98
health HEALTH_OK
monmap e1: 1 mons at {ceph-monitor=10.0.0.129:6789/0}
election epoch 3, quorum 0 ceph-monitor
fsmap e7: 1/1/1 up {0=ceph-osd=up:active}, 2 up:standby
osdmap e21: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v58: 320 pgs, 3 pools, 2068 bytes data, 20 objects
326 MB used, 45720 MB / 46046 MB avail
320 active+clean
[root@ceph-admin ceph]#其中，HEALTH_OK表示ceph集群正常
四、Kubernetes挂载ceph rbd存储卷
kubernetes是支持Ceph存储系统创建的块存储、存储文件系统等。kubernetes要想使用ceph，需要在k8s的每个节点安装ceph-common。
1. 安装ceph-common
#k8s环境节点
[root@k8s-master1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready control-plane,master 58d v1.20.6
k8s-node1 Ready worker 58d v1.20.6
k8s-node2 Ready worker 58d v1.20.6#每个节点配置ceph.repo源
[root@ceph-admin ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.131:/etc/yum.repos.d/ceph.repo
The authenticity of host '10.0.0.131 (10.0.0.131)' can't be established.
ECDSA key fingerprint is SHA256:O2leSOvudbcqIRBokjf4cUtbvjzdf/Yl49VkIQGfLxE.
ECDSA key fingerprint is MD5:de:41:d0:68:53:e3:08:09:b0:7a:55:2e:b6:1d:af:d3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.131' (ECDSA) to the list of known hosts.
root@10.0.0.131's password:
ceph.repo 100% 681 42.5KB/s 00:00
#k8s的控制节点上查看ceph.repo
[root@k8s-master1 ~]# ll /etc/yum.repos.d/ceph.repo
-rw-r--r-- 1 root root 681 Sep 28 21:16 /etc/yum.repos.d/ceph.repo
You have new mail in /var/spool/mail/root
[root@k8s-master1 ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.132:/etc/yum.repos.d/ceph.repo
ceph.repo 100% 681 511.5KB/s 00:00
[root@k8s-master1 ~]# scp /etc/yum.repos.d/ceph.repo 10.0.0.133:/etc/yum.repos.d/ceph.repo
ceph.repo 100% 681 338.9KB/s 00:00
[root@k8s-master1 ~]##每个节点上安装ceph-common
[root@k8s-master1 ~]# yum install ceph-common -y
[root@k8s-node1 ~]# yum install ceph-common -y
[root@k8s-node2 ~]# yum install ceph-common -y2. 拷贝ceph的配置文件
[root@ceph-admin ~]# scp /etc/ceph/* 10.0.0.131:/etc/ceph/
root@10.0.0.131's password:
ceph.bootstrap-mds.keyring 100% 113 0.2KB/s 00:00
ceph.bootstrap-mgr.keyring 100% 71 3.5KB/s 00:00
ceph.bootstrap-osd.keyring 100% 113 31.5KB/s 00:00
ceph.bootstrap-rgw.keyring 100% 113 1.0KB/s 00:00
ceph.client.admin.keyring 100% 129 84.2KB/s 00:00
ceph.conf 100% 292 136.0KB/s 00:00
ceph-deploy-ceph.log 100% 93KB 144.0KB/s 00:00
ceph.mon.keyring 100% 73 0.1KB/s 00:00
rbdmap 100% 92 1.5KB/s 00:00
[root@k8s-master1 ~]# scp /etc/ceph/* 10.0.0.132:/etc/ceph/
ceph.bootstrap-mds.keyring 100% 113 30.2KB/s 00:00
ceph.bootstrap-mgr.keyring 100% 71 31.0KB/s 00:00
ceph.bootstrap-osd.keyring 100% 113 80.5KB/s 00:00
ceph.bootstrap-rgw.keyring 100% 113 44.5KB/s 00:00
ceph.client.admin.keyring 100% 129 92.3KB/s 00:00
ceph.conf 100% 292 215.9KB/s 00:00
ceph-deploy-ceph.log 100% 93KB 29.5MB/s 00:00
ceph.mon.keyring 100% 73 49.9KB/s 00:00
rbdmap 100% 92 65.6KB/s 00:00
[root@k8s-master1 ~]# scp /etc/ceph/* 10.0.0.133:/etc/ceph/
ceph.bootstrap-mds.keyring 100% 113 82.1KB/s 00:00
ceph.bootstrap-mgr.keyring 100% 71 47.8KB/s 00:00
ceph.bootstrap-osd.keyring 100% 113 105.3KB/s 00:00
ceph.bootstrap-rgw.keyring 100% 113 75.9KB/s 00:00
ceph.client.admin.keyring 100% 129 83.4KB/s 00:00
ceph.conf 100% 292 246.0KB/s 00:00
ceph-deploy-ceph.log 100% 93KB 5.9MB/s 00:00
ceph.mon.keyring 100% 73 27.7KB/s 00:00
rbdmap 100% 92 84.8KB/s 00:00
[root@k8s-master1 ~]#3. 创建ceph rbd
[root@ceph-admin ~]# ceph osd pool create k8srbd1 6
pool 'k8srbd1' created
You have new mail in /var/spool/mail/root
[root@ceph-admin ~]# rbd create rbda -s 1024 -p k8srbd1
[root@ceph-admin ~]# rbd feature disable k8srbd1/rbda object-map fast-diff deep-flatten4. 创建pod，挂载ceph rbd
[root@k8s-master1 ~]# mkdir ceph
[root@k8s-master1 ~]# cd ceph/
[root@k8s-master1 ceph]# vim rbd-pod.yaml
[root@k8s-master1 ceph]# cat rbd-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: rbd-pod
spec:
containers:
- image: nginx:latest
name: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: testrbd
mountPath: /usr/share/nginx/html
volumes:
- name: testrbd
rbd:
monitors:
- '10.0.0.130:6789'
- '10.0.0.129:6789'
- '10.0.0.128:6789'
pool: k8srbd1
image: rbda
fsType: xfs
readOnly: false
user: admin
keyring: /etc/ceph/ceph.client.admin.keyring
[root@k8s-master1 ceph]# kubectl apply -f rbd-pod.yaml
pod/rbd-pod created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-5d65b75f7-2tv4q 1/1 Running 10 2d23h
rbd-pod 0/1 ContainerCreating 0 4s
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-5d65b75f7-2tv4q 1/1 Running 10 2d23h
rbd-pod 1/1 Running 0 47s
[root@k8s-master1 ceph]#rbd-pod已经创建成功，可以看到k8srbd1下的rbda被rbd-pod挂载了。通过以下命令，可以可能到rbd-pod的详情
[root@k8s-master1 ceph]# kubectl describe pods rbd-pod
Name: rbd-pod
Namespace: default
Priority: 0
Node: k8s-node1/10.0.0.132
Start Time: Wed, 28 Sep 2022 21:30:26 +0800
Labels: <none>
Annotations: cni.projectcalico.org/podIP: 10.244.36.99/32
cni.projectcalico.org/podIPs: 10.244.36.99/32
Status: Running
IP: 10.244.36.99
IPs:
IP: 10.244.36.99
Containers:
nginx:
Container ID: docker://e62cd5f3e715d1ad2f47cfbaf8b61583c700dd11cca03a884c637b11ad3047c3
Image: nginx:latest
Image ID: docker-pullable://nginx@sha256:b95a99feebf7797479e0c5eb5ec0bdfa5d9f504bc94da550c2f58e839ea6914f
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 28 Sep 2022 21:30:33 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from testrbd (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
testrbd:
Type: RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
CephMonitors: [10.0.0.130:6789 10.0.0.129:6789 10.0.0.128:6789]
RBDImage: rbda
FSType: xfs
RBDPool: k8srbd1
RadosUser: admin
Keyring: /etc/ceph/ceph.client.admin.keyring
SecretRef: nil
ReadOnly: false
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m8s default-scheduler Successfully assigned default/rbd-pod to k8s-node1
Normal SuccessfulAttachVolume 3m9s attachdetach-controller AttachVolume.Attach succeeded for volume "testrbd"
Normal Pulled 3m3s kubelet Container image "nginx:latest" already present on machine
Normal Created 3m3s kubelet Created container nginx
Normal Started 3m2s kubelet Started container nginx验证其他pod是否能还能挂载这个k8srbd1下的rbda，再创建一个rbd-pod1.yaml清单文件
[root@k8s-master1 ceph]# vim rbd-pod1.yaml
[root@k8s-master1 ceph]# cat rbd-pod1.yaml
apiVersion: v1
kind: Pod
metadata:
name: rbd-pod1
spec:
containers:
- image: nginx:latest
name: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: testrbd
mountPath: /usr/share/nginx/html
volumes:
- name: testrbd
rbd:
monitors:
- '10.0.0.130:6789'
- '10.0.0.129:6789'
- '10.0.0.128:6789'
pool: k8srbd1
image: rbda
fsType: xfs
readOnly: false
user: admin
keyring: /etc/ceph/ceph.client.admin.keyring
[root@k8s-master1 ceph]# kubectl apply -f rbd-pod1.yaml
pod/rbd-pod1 created
[root@k8s-master1 ceph]# kubectl get pods |grep rbd-pod
rbd-pod 1/1 Running 0 8m39s
rbd-pod1 0/1 ContainerCreating 0 11s
[root@k8s-master1 ceph]# kubectl get pods |grep rbd-pod
rbd-pod 1/1 Running 0 13m
rbd-pod1 0/1 ContainerCreating 0 5m14s可以看到rbd-pod1一直在创建中，使用kubectl describe pods rbd-pod1查看详细信息：
[root@k8s-master1 ~]# kubectl describe pods rbd-pod1
Name: rbd-pod1
Namespace: default
Priority: 0
Node: k8s-node2/10.0.0.133
Start Time: Wed, 28 Sep 2022 21:38:54 +0800
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
nginx:
Container ID:
Image: nginx:latest
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from testrbd (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
testrbd:
Type: RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
CephMonitors: [10.0.0.130:6789 10.0.0.129:6789 10.0.0.128:6789]
RBDImage: rbda
FSType: xfs
RBDPool: k8srbd1
RadosUser: admin
Keyring: /etc/ceph/ceph.client.admin.keyring
SecretRef: nil
ReadOnly: false
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 99s default-scheduler Successfully assigned default/rbd-pod1 to k8s-node2
Normal SuccessfulAttachVolume 99s attachdetach-controller AttachVolume.Attach succeeded for volume "testrbd"
Warning FailedMount 21s kubelet MountVolume.WaitForAttach failed for volume "testrbd" : rbd image k8srbd1/rbda is still being used通过warnning可以发现是因为pool: k8srbd1的rbda已经被其他pod占用了
五、基于ceph rbd生成pv
1. 创建ceph-secret
创建k8s secret对象ceph-secret，这个secret对象用于k8s volume插件访问ceph集群，获取client.admin的keyring值，并用base64编码，在ceph-admin（ceph管理节点）操作
[root@ceph-admin ~]# ceph auth get-key client.admin | base64
QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ==创建ceph的secret
[root@k8s-master1 ceph]# vim ceph-secret.yaml
[root@k8s-master1 ceph]# cat ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
data:
key: QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ==
[root@k8s-master1 ceph]# kubectl apply -f ceph-secret.yaml
secret/ceph-secret created
[root@k8s-master1 ceph]# kubectl get secret
NAME TYPE DATA AGE
ceph-secret Opaque 1 7s2. 创建pool池
[root@ceph-admin ~]# ceph osd pool create k8stest 6
pool 'k8stest' created
You have new mail in /var/spool/mail/root
[root@ceph-admin ~]# rbd create rbda -s 1024 -p k8stest
[root@ceph-admin ~]# rbd feature disable k8stest/rbda object-map fast-diff deep-flatten
[root@ceph-admin ~]# ceph osd lspools
0 rbd,1 cephfs_data,2 cephfs_metadata,3 k8srbd1,4 k8stest,3. 创建PV
[root@k8s-master1 ceph]# vim ceph-pv.yaml
[root@k8s-master1 ceph]# cat ceph-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: ceph-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
rbd:
monitors:
- '10.0.0.130:6789'
- '10.0.0.129:6789'
- '10.0.0.128:6789'
pool: k8stest
image: rbda
user: admin
secretRef:
name: ceph-secret
fsType: xfs
readOnly: false
persistentVolumeReclaimPolicy: Recycle
[root@k8s-master1 ceph]# kubectl apply -f ceph-pv.yaml
persistentvolume/ceph-pv created
[root@k8s-master1 ceph]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ceph-pv 1Gi RWO Recycle Available 4s4. 创建PVC
[root@k8s-master1 ceph]# vim ceph-pvc.yaml
[root@k8s-master1 ceph]# cat ceph-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ceph-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
[root@k8s-master1 ceph]# kubectl apply -f ceph-pvc.yaml
persistentvolumeclaim/ceph-pvc created
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-pvc Bound ceph-pv 1Gi RWO 5s创建的PVC自动与之前创建的ceph-pv绑定
5. 挂载PVC
[root@k8s-master1 ceph]# vim ceph-deploy.yaml
[root@k8s-master1 ceph]# cat ceph-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ceph-deploy
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/ceph-data"
name: ceph-data
volumes:
- name: ceph-data
persistentVolumeClaim:
claimName: ceph-pvc
[root@k8s-master1 ceph]# kubectl apply -f ceph-deploy.yaml
deployment.apps/ceph-deploy created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
ceph-deploy-5bb47d859f-chtpr 0/1 ContainerCreating 0 5s
ceph-deploy-5bb47d859f-fdp7w 0/1 ContainerCreating 0 5s
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
ceph-deploy-5bb47d859f-chtpr 1/1 Running 0 46s
ceph-deploy-5bb47d859f-fdp7w 1/1 Running 0 46s可以看到有deployment创建的两个pod正常运行，而且它们均调度到k8s-node1节点上，正常挂载ceph-pvc。所以可以发现ceph rbd块存储能在同一个node上跨pod以ReadWriteOnce共享挂载。
[root@k8s-master1 ceph]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ceph-deploy-5bb47d859f-chtpr 1/1 Running 0 4m38s 10.244.36.103 k8s-node1 <none> <none>
ceph-deploy-5bb47d859f-fdp7w 1/1 Running 0 4m38s 10.244.36.91 k8s-node1 <none> <none>再创建一个调度到不同节点上的pod，挂载ceph-pvc，测试是否能挂载成功
[root@k8s-master1 ceph]# vim ceph-deploy-test.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat ceph-deploy-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ceph-deploy-test
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: nginx
topologyKey: kubernetes.io/hostname
containers:
- name: nginx
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/ceph-data"
name: ceph-data
volumes:
- name: ceph-data
persistentVolumeClaim:
claimName: ceph-pvc
[root@k8s-master1 ceph]# kubectl apply -f ceph-deploy-test.yaml
deployment.apps/ceph-deploy-test created
[root@k8s-master1 ceph]# kubectl get pods -o wide |grep ceph-deploy-test
ceph-deploy-test-6c88bffc9d-rsgzk 0/1 ContainerCreating 0 113s <none> k8s-node2 <none> <none>可以看到调度到k8s-node2节点上的ceph-deploy-test-6c88bffc9d-rsgzk一直处于异常状态，通过以下命令查看详情发现，pod是挂载失败
[root@k8s-master1 ceph]# kubectl describe pods ceph-deploy-test-6c88bffc9d-rsgzk
Name: ceph-deploy-test-6c88bffc9d-rsgzk
Namespace: default
Priority: 0
Node: k8s-node2/10.0.0.133
Start Time: Wed, 28 Sep 2022 22:50:33 +0800
Labels: app=nginx
pod-template-hash=6c88bffc9d
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/ceph-deploy-test-6c88bffc9d
Containers:
nginx:
Container ID:
Image: nginx:latest
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/ceph-data from ceph-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
ceph-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: ceph-pvc
ReadOnly: false
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m27s default-scheduler Successfully assigned default/ceph-deploy-test-6c88bffc9d-rsgzk to k8s-node2
Warning FailedAttachVolume 4m27s attachdetach-controller Multi-Attach error for volume "ceph-pv" Volume is already used by pod(s) ceph-deploy-5bb47d859f-chtpr, ceph-deploy-5bb47d859f-fdp7w
Warning FailedMount 2m24s kubelet Unable to attach or mount volumes: unmounted volumes=[ceph-data], unattached volumes=[ceph-data default-token-5n29f]: timed out waiting for the condition
Warning FailedMount 8s kubelet Unable to attach or mount volumes: unmounted volumes=[ceph-data], unattached volumes=[default-token-5n29f ceph-data]: timed out waiting for the condition6. 总结
通过以上实验，可以得出ceph rbd块存储的特点：
1）ceph rbd块存储能在同一个node上同一个pod多个容器中以ReadWriteOnce共享挂载
2）ceph rbd块存储能在同一个node上跨pod以ReadWriteOnce共享挂载
3）ceph rbd块存储不能跨node以ReadWriteOnce共享挂载
如果一个使用ceph rdb的pod所在的node挂掉，这个pod虽然会被调度到其它node，但是由于rbd不能跨node多次挂载和挂掉的pod不能自动解绑pv的问题，这个新pod不会正常运行
解决办法：
1）使用能支持跨node和pod之间挂载的共享存储，例如cephfs，GlusterFS等
2）给node添加label，只允许deployment所管理的pod调度到一个固定的node上。（不建议，这个node挂掉的话，服务就故障了）
六、基于Cephfs创建pv
cephfs方式支持k8s的pv的3种访问模式ReadWriteOnce，ReadOnlyMany ，ReadWriteMany
1. 查看ceph文件系统
[root@ceph-admin ~]# ceph fs ls
name: test-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]2. 创建ceph子目录
为了能挂载cephfs，先创建一个secretfile
[root@ceph-admin ~]# cat /etc/ceph/ceph.client.admin.keyring |grep key|awk -F" " '{print $3}' > /etc/ceph/admin.secret
[root@ceph-admin ~]# cat /etc/ceph/admin.secret
AQCGETNjEpM+LhAAdBmi3lL7aSs2iQd49gcDXg==
[root@ceph-admin ~]#挂载cephfs的根目录到集群的mon节点下的一个目录，比如test_data，因为挂载后，就可以直接在test_data下面用Linux命令创建子目录
[root@ceph-admin ~]# mkdir test_data
You have new mail in /var/spool/mail/root
[root@ceph-admin ~]# mount -t ceph 10.0.0.129:6789:/ /root/test_data -o name=admin,secretfile=/etc/ceph/admin.secret
[root@ceph-admin ~]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 2.0G 12M 2.0G 1% /run
tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda2 xfs 40G 3.5G 37G 9% /
/dev/sda1 xfs 473M 140M 333M 30% /boot
/dev/sdb1 xfs 15G 123M 15G 1% /var/lib/ceph/osd/ceph-0
tmpfs tmpfs 394M 0 394M 0% /run/user/0
10.0.0.129:6789:/ ceph 45G 372M 45G 1% /root/test_data
[root@ceph-admin ~]#在cephfs的根目录里面创建了一个子目录lucky，k8s以后就可以挂载这个目录
[root@ceph-admin ~]# cd test_data/
You have new mail in /var/spool/mail/root
[root@ceph-admin test_data]# mkdir lucky
[root@ceph-admin test_data]# chmod 0777 lucky/
[root@ceph-admin test_data]# ll -a
total 0
drwxr-xr-x 1 root root 1 Sep 29 22:32 .
dr-xr-x---. 5 root root 216 Sep 29 22:29 ..
drwxrwxrwx 1 root root 0 Sep 29 22:32 lucky3. 创建Secret对象
获取client.admin的keyring值，并用base64编码
[root@ceph-admin test_data]# ceph auth get-key client.admin | base64
QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ==创建cephfs-secret这个secret对象，这个secret对象用于k8s volume插件访问cephfs文件系统
[root@k8s-master1 ceph]# vim cephfs-secret.yaml
[root@k8s-master1 ceph]# cat cephfs-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: cephfs-secret
data:
key: QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ==
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# kubectl apply -f cephfs-secret.yaml
secret/cephfs-secret created
[root@k8s-master1 ceph]# kubectl get secret |grep cephfs-secret
cephfs-secret Opaque 1 21s4. 创建PV
[root@k8s-master1 ceph]# vim cephfs-pv.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: cephfs-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
cephfs:
monitors:
- 10.0.0.129:6789
path: /lucky
user: admin
readOnly: false
secretRef:
name: cephfs-secret
persistentVolumeReclaimPolicy: Recycle
[root@k8s-master1 ceph]# kubectl apply -f cephfs-pv.yaml
persistentvolume/cephfs-pv created
[root@k8s-master1 ceph]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ceph-pv 1Gi RWO Recycle Bound default/ceph-pvc 24h
cephfs-pv 1Gi RWX Recycle Available 23s5. 创建PVC
[root@k8s-master1 ceph]# vim cephfs-pvc.yaml
[root@k8s-master1 ceph]# cat cephfs-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: cephfs-pvc
spec:
accessModes:
- ReadWriteMany
volumeName: cephfs-pv
resources:
requests:
storage: 1Gi
[root@k8s-master1 ceph]# kubectl apply -f cephfs-pvc.yaml
persistentvolumeclaim/cephfs-pvc created
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-pvc Bound ceph-pv 1Gi RWO 24h
cephfs-pvc Pending cephfs-pv 0 4s
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-pvc Bound ceph-pv 1Gi RWO 24h
cephfs-pvc Bound cephfs-pv 1Gi RWX 56s发现创建的cephfs-pvc已经自动与cephfs-pv绑定
6. 创建pod挂载cephfs-pvc
[root@k8s-master1 ceph]# vim cephfs-pod-1.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-pod-1.yaml
apiVersion: v1
kind: Pod
metadata:
name: cephfs-pod-1
spec:
containers:
- image: nginx:latest
name: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: test-v1
mountPath: /usr/share/nginx/html
volumes:
- name: test-v1
persistentVolumeClaim:
claimName: cephfs-pvc
[root@k8s-master1 ceph]# kubectl apply -f cephfs-pod-1.yaml
pod/cephfs-pod-1 created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
cephfs-pod-1 1/1 Running 0 79s登录到该pod中，执行以下命令：
[root@k8s-master1 ceph]# kubectl exec -it cephfs-pod-1 -- /bin/sh
# cd /usr/share/nginx/html
# ls
# echo "hello world" >> index.html
# echo "access to cephfs-pod-1" >>index.html
# cat index.html
hello world
access to cephfs-pod-1
# exit回到ceph-admin节点上，查看cephfs文件目录下是否有内容
[root@ceph-admin test_data]# ll lucky/
total 1
-rw-r--r-- 1 root root 35 Sep 29 22:52 index.html
[root@ceph-admin test_data]# cat lucky/index.html
hello world
access to cephfs-pod-1
[root@ceph-admin test_data]# pwd
/root/test_data再创建一个pod，挂载该cephfs-pvc卷
[root@k8s-master1 ceph]# vim cephfs-pod-2.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-pod-2.yaml
apiVersion: v1
kind: Pod
metadata:
name: cephfs-pod-2
spec:
containers:
- image: nginx
name: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: test-v2
mountPath: /usr/share/nginx/html
volumes:
- name: test-v2
persistentVolumeClaim:
claimName: cephfs-pvc
[root@k8s-master1 ceph]# kubectl apply -f cephfs-pod-2.yaml
pod/cephfs-pod-2 created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
cephfs-pod-1 1/1 Running 0 7m53s
cephfs-pod-2 1/1 Running 0 4s
nfs-client-provisioner-5d65b75f7-2tv4q 1/1 Running 12 4d1h
rbd-pod 1/1 Running 1 25h
[root@k8s-master1 ceph]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cephfs-pod-1 1/1 Running 0 8m11s 10.244.169.164 k8s-node2 <none> <none>
cephfs-pod-2 1/1 Running 0 22s 10.244.36.109 k8s-node1 <none> <none>访问cephfs-pod-2,并登录到该pod容器中，查看/usr/share/nginx/html该目录下是否有文件
[root@k8s-master1 ceph]# curl 10.244.36.109
hello world
access to cephfs-pod-1
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# kubectl exec -it cephfs-pod-2 -- /bin/sh
# ls /usr/share/nginx/html
index.html
# cat /usr/share/nginx/html/index.html
hello world
access to cephfs-pod-1
# exit
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]#从上述实验中可以看出：调度到不同节点上的cephfs-pod-1和cephfs-pod-2可以共享cephfs-pvc卷
七、使用Ceph实现动态持久卷存储
动态供给能够自动帮忙创建pv，需要多大的空间就创建多大的pv。k8s帮助创建pv，创建pvc就直接api调用存储类来寻找pv。
如果是存储静态供给的话，需要手动去创建pv，如果没有足够的资源，找不到合适的pv，那么pod就会处于pending等待的状态。而动态供给主要的一个实现就是StorageClass存储对象，其实它就是声明使用哪个存储，然后帮忙去连接，再自动创建pv。
1. 使用Ceph RBD作为持久数据卷
1） 为Ceph配置文件赋权
在所有节点，ceph集群节点和k8s集群节点执行赋权操作：
[root@k8s-master1 ~]# chmod 777 -R /etc/ceph/*
[root@k8s-master1 ~]# mkdir /root/.ceph/
[root@k8s-master1 ~]# cp -ar /etc/ceph/ /root/.ceph/其他节点与k8s-master1节点操作相同
2）配置 rbd-provisioner
[root@k8s-master1 ceph]# vim external-storage-rbd-provisioner.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat external-storage-rbd-provisioner.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns","coredns"]
verbs: ["list", "get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: rbd-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-provisioner
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-provisioner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rbd-provisioner
spec:
selector:
matchLabels:
app: rbd-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: rbd-provisioner
spec:
containers:
- name: rbd-provisioner
image: quay.io/external_storage/rbd-provisioner:latest
imagePullPolicy: IfNotPresent
env:
- name: PROVISIONER_NAME
value: ceph.com/rbd
serviceAccount: rbd-provisioner
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-provisioner
[root@k8s-master1 ceph]# kubectl apply -f external-storage-rbd-provisioner.yaml
clusterrole.rbac.authorization.k8s.io/rbd-provisioner created
clusterrolebinding.rbac.authorization.k8s.io/rbd-provisioner created
role.rbac.authorization.k8s.io/rbd-provisioner created
rolebinding.rbac.authorization.k8s.io/rbd-provisioner created
deployment.apps/rbd-provisioner created
serviceaccount/rbd-provisioner created
[root@k8s-master1 ceph]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 0 11s 10.244.36.114 k8s-node1 <none> <none>3）创建ceph-secret
#创建pool池
[root@ceph-admin ~]# ceph osd pool create k8stest1 6
pool 'k8stest1' created
[root@ceph-admin ~]# ceph osd lspools
0 rbd,1 cephfs_data,2 cephfs_metadata,3 k8srbd1,4 k8stest,5 k8stest1,创建secret资源
[root@k8s-master1 ceph]# vim ceph-secret-1.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat ceph-secret-1.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret-1
type: "ceph.com/rbd"
data:
key: QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ==
[root@k8s-master1 ceph]# kubectl apply -f ceph-secret-1.yaml
secret/ceph-secret-1 created
[root@k8s-master1 ceph]# kubectl get secret
NAME TYPE DATA AGE
ceph-secret-1 ceph.com/rbd 1 12s
default-token-5n29f kubernetes.io/service-account-token 3 60d
rbd-provisioner-token-qbtmt kubernetes.io/service-account-token 3 11m4）配置 storageclass
[root@k8s-master1 ceph]# vim rbd-storageclass.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat rbd-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rbd-storageclass
provisioner: ceph.com/rbd
parameters:
monitors: 10.0.0.130:6789,10.0.0.129:6789,10.0.0.128:6789
adminId: admin
adminSecretName: ceph-secret-1
pool: k8stest1
userId: admin
userSecretName: ceph-secret-1
fsType: xfs
imageFormat: "2"
imageFeatures: "layering"
[root@k8s-master1 ceph]# kubectl apply -f rbd-storageclass.yaml
storageclass.storage.k8s.io/rbd-storageclass created
[root@k8s-master1 ceph]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rbd-storageclass ceph.com/rbd Delete Immediate false 6s5）创建PVC
[root@k8s-master1 ceph]# vim rbd-pvc.yaml
[root@k8s-master1 ceph]# cat rbd-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: rbd-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: rbd-storageclass
[root@k8s-master1 ceph]# kubectl apply -f rbd-pvc.yaml
persistentvolumeclaim/rbd-pvc created
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
rbd-pvc Pending rbd-storageclass 6s发现创建的rbd-pvc一直处于pending状态，用以下命令查看pvc详情
[root@k8s-master1 ceph]# kubectl describe pvc rbd-pvc
Name: rbd-pvc
Namespace: default
StorageClass: rbd-storageclass
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: ceph.com/rbd
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 11s (x3 over 30s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ceph.com/rbd" or manually created by system administrator发现找不到external provisioner "ceph.com/rbd"，查看rbd-provisioner-65bf77fd59-9jnm8 日志
[root@k8s-master1 ceph]# kubectl logs rbd-provisioner-65bf77fd59-9jnm8
I0930 13:26:50.323553 1 main.go:85] Creating RBD provisioner ceph.com/rbd with identity: ceph.com/rbd
I0930 13:26:50.325012 1 leaderelection.go:185] attempting to acquire leader lease default/ceph.com-rbd...
E0930 13:26:50.336351 1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"ceph.com-rbd", GenerateName:"", Namespace:"default", SelfLink:"", UID:"e2fc1dc4-aa04-4667-b179-051d1b2721cb", ResourceVersion:"544566", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63800141210, loc:(*time.Location)(0x1bc94e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"rbd-provisioner-65bf77fd59-9jnm8_8a68381c-40c3-11ed-b978-befb50ef5a7c\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2022-09-30T13:26:50Z\",\"renewTime\":\"2022-09-30T13:26:50Z\",\"leaderTransitions\":0}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'rbd-provisioner-65bf77fd59-9jnm8_8a68381c-40c3-11ed-b978-befb50ef5a7c became leader'
I0930 13:26:50.336551 1 leaderelection.go:194] successfully acquired lease default/ceph.com-rbd
I0930 13:26:50.336868 1 controller.go:631] Starting provisioner controller ceph.com/rbd_rbd-provisioner-65bf77fd59-9jnm8_8a68381c-40c3-11ed-b978-befb50ef5a7c!
I0930 13:26:50.438154 1 controller.go:680] Started provisioner controller ceph.com/rbd_rbd-provisioner-65bf77fd59-9jnm8_8a68381c-40c3-11ed-b978-befb50ef5a7c!
I0930 13:49:52.814754 1 controller.go:987] provision "default/rbd-pvc" class "rbd-storageclass": started
E0930 13:49:52.822329 1 controller.go:1004] provision "default/rbd-pvc" class "rbd-storageclass": unexpected error getting claim reference: selfLink was empty, can't make reference这是因为使用的是1.20版本的k8s集群，它通过rbd provisioner动态生成pv会报如上错误。解决方法：修改/etc/kubernetes/manifests/kube-apiserver.yaml配置文件，添加- --feature-gates=RemoveSelfLink=false 一行这样的参数，如下：
添加完成后，保存退出。重启kubelet服务，查看kube-system空间下的组件正常运行，即可。
[root@k8s-master1 ceph]# systemctl restart kubelet
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Fri 2022-09-30 22:00:44 CST; 8s ago
Docs: https://kubernetes.io/docs/
Main PID: 71245 (kubelet)
Memory: 43.3M
CGroup: /system.slice/kubelet.service
└─71245 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kub...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.485710 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...2156438")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.485815 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...2156438")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486011 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...b621314")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486158 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume ...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486238 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume ...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486316 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...5c91746")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486405 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume ...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486485 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started fo...b621314")
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486560 71245 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume ...
Sep 30 22:00:52 k8s-master1 kubelet[71245]: I0930 22:00:52.486634 71245 reconciler.go:157] Reconciler: start to sync state
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-master1 ceph]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6949477b58-82r4z 1/1 Running 24 60d
calico-node-kr8bt 1/1 Running 35 60d
calico-node-pzzlf 1/1 Running 38 60d
calico-node-wwrjq 1/1 Running 53 60d
coredns-7f89b7bc75-2cgxw 1/1 Running 25 60d
coredns-7f89b7bc75-gm6jp 1/1 Running 25 60d
etcd-k8s-master1 1/1 Running 24 60d
kube-apiserver-k8s-master1 1/1 Running 0 4m35s
kube-controller-manager-k8s-master1 1/1 Running 88 60d
kube-proxy-4tnzv 1/1 Running 24 60d
kube-proxy-mnnsg 1/1 Running 24 60d
kube-proxy-mxnhb 1/1 Running 26 60d
kube-scheduler-k8s-master1 1/1 Running 84 60d
[root@k8s-master1 ceph]#再次查看之前创建的rbd-pvc，已显示正常绑定pv
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
rbd-pvc Bound pvc-459cb59a-8504-4800-9264-388f3a7dc397 1Gi RWO rbd-storageclass 13m
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-459cb59a-8504-4800-9264-388f3a7dc397 1Gi RWO Delete Bound default/rbd-pvc rbd-storageclass 5m17s6）创建pod挂载PVC
[root@k8s-master1 ceph]# vim rbd-sc-nginx-pod.yaml
[root@k8s-master1 ceph]# cat rbd-sc-nginx-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: rbd-sc-nginx-pod
labels:
name: rbd-sc-nginx-pod
spec:
containers:
- name: rbd-sc-nginx-pod
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 80
volumeMounts:
- name: ceph-rdb
mountPath: /usr/share/nginx/html
volumes:
- name: ceph-rdb
persistentVolumeClaim:
claimName: rbd-pvc
[root@k8s-master1 ceph]# kubectl apply -f rbd-sc-nginx-pod.yaml
pod/rbd-sc-nginx-pod created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 42m
rbd-sc-nginx-pod 0/1 ContainerCreating 0 5s
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 43m
rbd-sc-nginx-pod 1/1 Running 0 66s
[root@k8s-master1 ceph]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 44m 10.244.36.114 k8s-node1 <none> <none>
rbd-sc-nginx-pod 1/1 Running 0 111s 10.244.36.116 k8s-node1 <none> <none>7）测试访问
[root@k8s-master1 ceph]# kubectl exec -it rbd-sc-nginx-pod -- /bin/sh -c 'echo Hello World from Ceph RBD!!! > /usr/share/nginx/html/index.html'
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# curl 10.244.36.116
Hello World from Ceph RBD!!!
[root@k8s-master1 ceph]#8)清理
[root@k8s-master1 ceph]# kubectl delete pods rbd-sc-nginx-pod
pod "rbd-sc-nginx-pod" deleted
[root@k8s-master1 ceph]# kubectl delete pvc rbd-pvc
persistentvolumeclaim "rbd-pvc" deleted
[root@k8s-master1 ceph]# kubectl get pvc
No resources found in default namespace.
[root@k8s-master1 ceph]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-459cb59a-8504-4800-9264-388f3a7dc397 1Gi RWO Delete Released default/rbd-pvc rbd-storageclass 19m
[root@k8s-master1 ceph]# kubectl get pv
No resources found
[root@k8s-master1 ceph]#可以看到pvc删除后，pv也会自动删除
2. 使用CephFS作为持久数据卷
1）查看当前CephFS
[root@ceph-admin ~]# ceph fs ls
name: test-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]可以看到已经创建了ceph文件系统test_cephfs，存储数据池为cephfs_data，存储元数据池为cephfs_metadata
若显示No filesystems enabled，则使用以下命令创建，创建两个pool分别存储数据和元数据，然后在创建CephFS
ceph osd pool create fs_data 8
ceph osd pool create fs_metadata 8
ceph fs new cephfs_name fs_metadata fs_data2）配置 cephfs-provisioner
[root@k8s-master1 ceph]# vim external-storage-cephfs-provisioner.yaml
[root@k8s-master1 ceph]# cat external-storage-cephfs-provisioner.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephfs-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cephfs-provisioner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cephfs-provisioner
subjects:
- kind: ServiceAccount
name: cephfs-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: cephfs-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cephfs-provisioner
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cephfs-provisioner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cephfs-provisioner
subjects:
- kind: ServiceAccount
name: cephfs-provisioner
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cephfs-provisioner
spec:
selector:
matchLabels:
app: cephfs-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: cephfs-provisioner
spec:
containers:
- name: cephfs-provisioner
image: "registry.cn-chengdu.aliyuncs.com/ives/cephfs-provisioner:latest"
imagePullPolicy: IfNotPresent
env:
- name: PROVISIONER_NAME
value: ceph.com/cephfs
command:
- "/usr/local/bin/cephfs-provisioner"
args:
- "-id=cephfs-provisioner-1"
serviceAccount: cephfs-provisioner
[root@k8s-master1 ceph]# kubectl apply -f external-storage-cephfs-provisioner.yaml
serviceaccount/cephfs-provisioner created
clusterrole.rbac.authorization.k8s.io/cephfs-provisioner created
clusterrolebinding.rbac.authorization.k8s.io/cephfs-provisioner created
role.rbac.authorization.k8s.io/cephfs-provisioner created
rolebinding.rbac.authorization.k8s.io/cephfs-provisioner created
deployment.apps/cephfs-provisioner created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
cephfs-provisioner-78c5545448-mtnbz 1/1 Running 0 6s
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 57m3）创建cephfs-secret
[root@k8s-master1 ceph]# vim cephfs-secret-1.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-secret-1.yaml
apiVersion: v1
kind: Secret
metadata:
name: cephfs-secret-1
type: "ceph.com/cephfs"
data:
key: QVFDR0VUTmpFcE0rTGhBQWRCbWkzbEw3YVNzMmlRZDQ5Z2NEWGc9PQ==
[root@k8s-master1 ceph]# kubectl apply -f cephfs-secret-1.yaml
secret/cephfs-secret-1 created
[root@k8s-master1 ceph]# kubectl get secrets
NAME TYPE DATA AGE
ceph-secret-1 ceph.com/rbd 1 49m
cephfs-provisioner-token-ltvpd kubernetes.io/service-account-token 3 3m41s
cephfs-secret-1 ceph.com/cephfs 1 9s
default-token-5n29f kubernetes.io/service-account-token 3 60d
rbd-provisioner-token-qbtmt kubernetes.io/service-account-token 3 61m4）配置 storageclass
[root@k8s-master1 ceph]# vim cephfs-storageclass.yaml
[root@k8s-master1 ceph]# cat cephfs-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: cephfs-storageclass
provisioner: ceph.com/cephfs
parameters:
monitors: 10.0.0.130:6789,10.0.0.129:6789,10.0.0.128:6789
adminId: admin
adminSecretName: cephfs-secret-1
claimRoot: /volumes/kubernetes
[root@k8s-master1 ceph]# kubectl apply -f cephfs-storageclass.yaml
storageclass.storage.k8s.io/cephfs-storageclass created
[root@k8s-master1 ceph]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
cephfs-storageclass ceph.com/cephfs Delete Immediate false 4s
rbd-storageclass ceph.com/rbd Delete Immediate false 49m5）创建PVC
[root@k8s-master1 ceph]# vim cephfs-sc-pvc.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-sc-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: cephfs-sc-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: cephfs-storageclass
resources:
requests:
storage: 2Gi
[root@k8s-master1 ceph]# kubectl apply -f cephfs-sc-pvc.yaml
persistentvolumeclaim/cephfs-sc-pvc created
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cephfs-sc-pvc Pending cephfs-storageclass 5s
[root@k8s-master1 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cephfs-sc-pvc Bound pvc-54b09482-965a-445c-8232-c99127e8839b 2Gi RWX cephfs-storageclass 17s
[root@k8s-master1 ceph]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-54b09482-965a-445c-8232-c99127e8839b 2Gi RWX Delete Bound default/cephfs-sc-pvc cephfs-storageclass 7s创建的cephfs-sc-pvc已经自动与pv绑定
6）创建pod挂载PVC
[root@k8s-master1 ceph]# vim cephfs-sc-nginx-pod.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 ceph]# cat cephfs-sc-nginx-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: cephfs-sc-nginx-pod
labels:
name: cephfs-sc-nginx-pod
spec:
containers:
- name: cephfs-sc-nginx-pod
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 80
volumeMounts:
- name: cephfs-data
mountPath: /usr/share/nginx/html
volumes:
- name: cephfs-data
persistentVolumeClaim:
claimName: cephfs-sc-pvc
[root@k8s-master1 ceph]# kubectl apply -f cephfs-sc-nginx-pod.yaml
pod/cephfs-sc-nginx-pod created
[root@k8s-master1 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
cephfs-provisioner-78c5545448-mtnbz 1/1 Running 0 17m
cephfs-sc-nginx-pod 0/1 ContainerCreating 0 6s
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 75m
[root@k8s-master1 ceph]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cephfs-provisioner-78c5545448-mtnbz 1/1 Running 0 17m 10.244.36.117 k8s-node1 <none> <none>
cephfs-sc-nginx-pod 1/1 Running 0 40s 10.244.36.121 k8s-node1 <none> <none>
rbd-provisioner-65bf77fd59-9jnm8 1/1 Running 2 75m 10.244.36.114 k8s-node1 <none> <none>7）测试访问
[root@k8s-master1 ceph]# kubectl exec -it cephfs-sc-nginx-pod -- /bin/sh -c 'echo Hello World from CephFS!!! > /usr/share/nginx/html/index.html'
[root@k8s-master1 ceph]# curl 10.244.36.121
Hello World from CephFS!!!
[root@k8s-master1 ceph]#
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
