简介
Ceph支持为云平台提供对象存储或者块设备服务、部署文件系统或其他目的。一个Ceph集群至少需要一个Ceph Monitor、Ceph Manager和Ceph OSD(Object Storage Daemon)。运行Ceph文件系统客户端时需要Ceph元数据服务
Monitors:Ceph Monitor(ceph-mon)维护集群状态的映射，包括monitor映射、manager映射、OSD映射、MDS映射和CRUSH映射。这些映射时Ceph守护进程相互协调所需的关键集群状态。monitor还负责管理守护进程和客户端之间的身份验证。通常需要三个monitor以上才能实现冗余和高可用性
Managers:Ceph Managers守护进程(ceph-mgr)负责跟踪运行时指标和Ceph集群的当前状态，包括存储利用率、当前性能指标和系统负载。Ceph Manager守护进程还托管基于python的模块来管理和公开Ceph集群信息，包括基于web的Ceph Dashboard和REST API。高可用性通常至少需要两个Managers
Ceph OSDs:对象存储守护进程(ceph-osd)存储数据，处理数据复制，恢复，重新平衡，并通过检查其他Ceph OSD守护进程的心跳向Ceph Monitors和Managers提供一些监控信息。通常至少需要三个Ceph OSD才能实现冗余和高可用性
MDSs:Ceph元数据服务器(ceph-mds)代表Ceph文件系统存储元数据(即Ceph块设备和Ceph对象存储)
Ceph架构
每个组件可扩展(空间能轻松扩展的PB级别)
不存在单点故障(高可靠)
可以运行在常规硬件上
Ceph集群部署
完全手动编译安装:
安装必要的依赖
下载源码包
手动编写ceph.conf文件
测试集群状态借助ceph-deploy、saltstack、ansible等工具部署
ceph-deploy方式部署
操作系统:centos7
网卡:双网卡，存储网络，管理网络
节点数量:3(3个osd，1个mon)
硬盘数量:2(1个日志，1个数据盘)
节点IP信息:
192.168.227.71 ceph-node1
192.168.227.72 ceph-node2
192.168.227.73 ceph-node3
hostnamectl set-hostname ceph-node1
echo "192.168.227.71 ceph-node1 ceph_node1
192.168.227.72 ceph-node2 ceph_node2
192.168.227.73 ceph-node3 ceph_node3" >> /etc/hostsceph orch daemon add osd ceph-node2:/dev/sdb
ceph orch
使用cephadmin安装
centos8
rpm -ivh https://mirrors.ustc.edu.cn/epel/8/Everything/x86_64/Packages/e/epel-release-8-19.el8.noarch.rpm
yum install ceph podman-docker python3 lvm2 -y
dnf search release-ceph
dnf install --assumeyes centos-release-ceph-reef ##centos-release-ceph-quincy
dnf install --assumeyes cephadm
cephadm install ceph-common
引导一个新集群
创建新 Ceph 集群的第一步是在 Ceph 集群的第一台主机上运行命令。在 Ceph 集群的第一台主机上运行该命令的行为 会创建 Ceph 集群的第一个“监控守护进程”，并且该监控守护进程需要一个 IP 地址。您必须将 Ceph 集群的第一台主机的 IP 地址传递给命令，因此您需要知道该主机的 IP 地址。
cephadm bootstrap --mon-ip *<mon-ip>*
[root@ceph_node1 ~]# cephadm bootstrap --mon-ip 192.168.227.71
该命令将：
1、在本地主机上为新集群创建监视器和管理器守护程序。
2、为 Ceph 集群生成新的 SSH 密钥并将其添加到 root 用户的/root/.ssh/authorized_keys文件中。
3、将公钥的副本写入/etc/ceph/ceph.pub。
4、将最小配置文件写入/etc/ceph/ceph.conf. 需要此文件与新集群进行通信。
5、client.admin将管理（特权！）密钥的副本写入/etc/ceph/ceph.client.admin.keyring。
6、将_admin标签添加到引导主机。/etc/ceph/ceph.conf默认情况下，任何具有此标签的主机都将（也）获得和 的副本/etc/ceph/ceph.client.admin.keyring。
URL: https://ceph-node1:8443/
User: admin
Password: vdeqc6nx11
[root@ceph_node1 ~]# cephadm bootstrap --mon-ip 192.168.227.71
Creating directory /etc/ceph for ceph.conf
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 4.6.1 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: dc5452d2-bb53-11ee-a6f0-00155d1f4246
Verifying IP 192.168.227.71 port 3300 ...
Verifying IP 192.168.227.71 port 6789 ...
Mon IP `192.168.227.71` is in CIDR network `192.168.227.0/24`
Mon IP `192.168.227.71` is in CIDR network `192.168.227.0/24`
Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image quay.io/ceph/ceph:v18...
Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
firewalld ready
Enabling firewalld service ceph-mon in current zone...
Waiting for mon to start...
Waiting for mon...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (1/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (2/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (3/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (4/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (5/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (6/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (7/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (8/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (9/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (10/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (11/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (12/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (13/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (14/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
mon not available, waiting (15/15)...
/usr/bin/ceph: timeout after 60 seconds
Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=ceph_node1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dc5452d2-bb53-11ee-a6f0-00155d1f4246/mon.ceph_node1:/var/lib/ceph/mon/ceph-ceph_node1:z -v /tmp/ceph-tmp2xmdiax3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw6qkop7z:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 status
Error: mon not available after 15 tries
***************
Cephadm hit an issue during cluster installation. Current cluster files will NOT BE DELETED automatically to change
this behaviour you can pass the --cleanup-on-failure. To remove this broken cluster manually please run:
> cephadm rm-cluster --force --fsid dc5452d2-bb53-11ee-a6f0-00155d1f4246
in case of any previous broken installation user must use the rm-cluster command to delete the broken cluster:
> cephadm rm-cluster --force --zap-osds --fsid <fsid>
for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster
***************
ERROR: mon not available after 15 tries
cephadm rm-cluster --force --fsid dc5452d2-bb53-11ee-a6f0-00155d1f4246
cephadm rm-cluster --force --zap-osds --fsid dc5452d2-bb53-11ee-a6f0-00155d1f4246
[root@ceph_node1 ~]# cephadm bootstrap --mon-ip 192.168.227.71
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 4.6.1 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: bd1270f8-bb61-11ee-be68-00155d1f4246
Verifying IP 192.168.227.71 port 3300 ...
OSError: [Errno 99] Cannot assign requested address
***************
Cephadm hit an issue during cluster installation. Current cluster files will NOT BE DELETED automatically to change
this behaviour you can pass the --cleanup-on-failure. To remove this broken cluster manually please run:
> cephadm rm-cluster --force --fsid bd1270f8-bb61-11ee-be68-00155d1f4246
in case of any previous broken installation user must use the rm-cluster command to delete the broken cluster:
> cephadm rm-cluster --force --zap-osds --fsid <fsid>
for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster
***************
Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/sbin/cephadm/__main__.py", line 10700, in <module>
File "/usr/sbin/cephadm/__main__.py", line 10688, in main
File "/usr/sbin/cephadm/__main__.py", line 6156, in _rollback
File "/usr/sbin/cephadm/__main__.py", line 2495, in _default_image
File "/usr/sbin/cephadm/__main__.py", line 6262, in command_bootstrap
File "/usr/sbin/cephadm/__main__.py", line 5522, in prepare_mon_addresses
File "/usr/sbin/cephadm/__main__.py", line 1693, in check_ip_port
File "/usr/sbin/cephadm/__main__.py", line 1644, in attempt_bind
File "/usr/sbin/cephadm/__main__.py", line 1637, in attempt_bind
OSError: [Errno 99] Cannot assign requested address
podman image rm quay.io/ceph/ceph:v18 [root@ceph-node1 ~]# podman image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/ceph/ceph v18 7f099bcd7014 4 weeks ago 1.28 GB
[root@ceph-node1 ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
908283df6120 quay.io/ceph/ceph:v18 -n mon.ceph_node1... 25 minutes ago Up 25 minutes ceph-9c903c22-c32f-11ee-a0cd-00155d1f4246-mon-ceph_node1
ba527bf12f52 quay.io/ceph/ceph:v18 -n mgr.ceph_node1... 25 minutes ago Up 25 minutes ceph-9c903c22-c32f-11ee-a0cd-00155d1f4246-mgr-ceph_node1-btosat
[root@ceph-node1 ~]#
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node1
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node3
ceph config set mon public_network 192.168.227.0/24
ceph orch host add ceph-node2 192.168.227.72 --labels=_admin
ceph orch host add ceph-node3 192.168.227.73 --labels=_admin
ceph orch host label add *<host>* _admin
ceph config set mon public_network 192.168.227.0/24
ceph orch daemon add mon ceph-node2:192.168.227.72
ceph orch daemon add mon ceph-node3:192.168.227.73
ceph orch apply mon ceph-node1,ceph-node2,ceph-node3
ceph orch ls --service_type mgr
ceph orch ps --daemon_type mgr
ceph orch apply osd --all-available-devices 全部
ceph orch daemon add osd ceph-node1:/dev/sdb
ceph orch daemon add osd ceph-node2:/dev/sdb
ceph orch daemon add osd ceph-node3:/dev/sdb
173 ceph orch osd tree
174 ceph orch osd ls
175 ceph osd tree
176 ceph osd ps
177 ceph osd ls
178 ceph osd tree
ceph fs volume create <fs_name> --placement="<placement spec>"
180 ceph fs volume create zhnfs
ceph fs volume info zhnfs
ceph osd pool stats
ceph fs authorize zhnfs
ceph fs authorize cephfs client.foo1 / rw
225 mount -t ceph :/ /mnt/cephfs -o name=admin,secret=AQBYSjZfQF+UJBAAC6QJjNACndkw2LcCR2XLFA==
226 mount -t ceph :/ /mnt -o name=admin,secret=AQBYSjZfQF+UJBAAC6QJjNACndkw2LcCR2XLFA==
227 mkdir /mnt/
228 mount -t ceph :/ /mnt -o name=admin,secret=AQBYSjZfQF+UJBAAC6QJjNACndkw2LcCR2XLFA==
229 mount -t ceph :/ /mnt
查看osd状态
ceph osd tree
1.删除对应osd
ceph osd crush reweight osd.1 0.1
2.停止osd进程
systemctl stop osd.1
3.将节点状态标记为out
ceph osd out osd.1
4.从crush中移除节点
ceph osd crush remove osd.1
5.删除节点
ceph osd rm osd.1
6.删除节点认证
ceph auth del osd.1
准备并激活osd
1.格式化硬盘
mkfs.ext4 /dev/sdb
2.挂载指定硬盘
mount /dev/sdb /var/lib/ceph/osd/ceph-1
3.准备osd
ceph-volume lvm prepare --data /dav/sdb
4.激活osd
ceph-volume lvm activce $id $fsid
激活后若还有down 执行以下命令重启并观察
systemctl | grep ceph
systemctl restart ceph-osd@1.service
ceph osd tree
COMMENTS | NOTHING