Linux搭建Ceph集群--详细流程介绍

近期在linux上搭建了用于分布式存储的----GlusterFS和Ceph这两个开源的分布式文件系统。

前言----大家可以去github上搜索一下，看源码或者官方文档介绍，更多的去了解Ceph，在这里我就不一一的去介绍原理以及抽象技术层面的基础知识。下面我就搭建部署过程中遇到的问题，向大家做一个介绍及部署过程的详细流程。同时，也希望研究或者喜好这方面的同学，带有生产环境成熟方案或者意见的，也请在这里互相讨论，我们一起共同进步，感兴趣的同学可以加我微信(请备注请求与来源)，互相沟通交流。

下面开始Ceph的详细介绍：遇到的问题也会在过程中解决，并且特别说明的是，安装中没有特别说明的，都是安装在同一台admin-node上的配置。

1.首先要查看你的网络是否与外网连通，好多都是虚拟机，需要配置代理，代理一定要配置在

/etc/environment 中，否则不起作用。或者在yum.conf中配置代理也可以。我选在/etc/environment中配置。

2.查看host及hostname名称

[root@vm-10-112-178-135 gadmin]# vi /etc/hosts

[root@vm-10-112-178-135 gadmin]# vi /etc/hostname

3.安装epel

[root@vm-10-112-178-135 gadmin]# yum install -y epel-release

已加载插件：fastestmirrorLoading mirror speeds from cached hostfile正在解决依赖关系--> 正在检查事务---> 软件包 epel-release.noarch.0.7-9 将被 安装--> 解决依赖关系完成依赖关系解决============================================================================================================================================================================================================================================ Package                                                        架构                                                    版本                                                  源                                                      大小============================================================================================================================================================================================================================================正在安装: epel-release                                                  noarch                                                  7-9                                                  epel                                                    14 k事务概要============================================================================================================================================================================================================================================安装  1 软件包总下载量：14 k安装大小：24 kDownloading packages:警告：/var/cache/yum/x86_64/7/epel/packages/epel-release-7-9.noarch.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID 352c64e5: NOKEYepel-release-7-9.noarch.rpm 的公钥尚未安装epel-release-7-9.noarch.rpm                                                                                                                                                                                          |  14 kB  00:00:00从 http://file.idc.pub/os/epel/RPM-GPG-KEY-EPEL-7 检索密钥导入 GPG key 0x352C64E5: 用户ID    : "Fedora EPEL (7)"

指纹      : 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5

来自      : http://file.idc.pub/os/epel/RPM-GPG-KEY-EPEL-7

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

正在安装    : epel-release-7-9.noarch                                                                                                        验证中      : epel-release-7-9.noarch                                                                                                                                                                                                1/1

已安装:

epel-release.noarch 0:7-9

完毕！

[root@vm-10-112-178-135 gadmin]# yum info epel-release

已加载插件：fastestmirror

Repository epel-testing is listed more than once in the configuration

Repository epel-testing-debuginfo is listed more than once in the configuration

Repository epel-testing-source is listed more than once in the configuration

Repository epel is listed more than once in the configuration

Repository epel-debuginfo is listed more than once in the configuration

Repository epel-source is listed more than once in the configuration

Loading mirror speeds from cached hostfile

已安装的软件包

名称    ：epel-release

架构    ：noarch

版本    ：7

发布    ：9

大小    ：24 k

源    ：installed

来自源：epel

简介    ： Extra Packages for Enterprise Linux repository configuration

网址    ：http://download.fedoraproject.org/pub/epel

协议    ： GPLv2

描述    ： This package contains the Extra Packages for Enterprise Linux (EPEL) repository

: GPG key as well as configuration for yum.

4.手动添加ceph.repo 的 yum资源库文件，下载ceph包文件从这里面下载

[root@vm-10-112-178-135 gadmin]# vi /etc/yum.repos.d/ceph.repo

文件内容，将以下文件内容复制到文件中：

[ceph]

name=Ceph packages for $basearch

baseurl=http://download.ceph.com/rpm-jewel/el7/$basearch

enabled=1

priority=1

gpgcheck=1

type=rpm-md

gpgkey=http://download.ceph.com/keys/release.asc

[ceph-noarch]

name=Ceph noarch packages

baseurl=http://download.ceph.com/rpm-jewel/el7/noarch

enabled=1

priority=1

gpgcheck=1

type=rpm-md

gpgkey=http://download.ceph.com/keys/release.asc

[ceph-x86_64]

name=Ceph x86_64 packages

baseurl=http://download.ceph.com/rpm-jewel/el7/x86_64

enabled=0

priority=1

gpgcheck=1

type=rpm-md

gpgkey=http://download.ceph.com/keys/release.asc

[ceph-aarch64]

name=Ceph source packages

baseurl=http://download.ceph.com/rpm-jewel/el7/aarch64

enabled=0

priority=1

gpgcheck=1

type=rpm-md

gpgkey=http://download.ceph.com/keys/release.asc

[ceph-source]

name=Ceph source packages

baseurl=http://download.ceph.com/rpm-jewel/el7/SRPMS

enabled=0

priority=1

gpgcheck=1

type=rpm-md

gpgkey=http://download.ceph.com/keys/release.asc

[apache2-ceph-noarch]

name=Apache noarch packages for Ceph

baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/master/SRPMS

enabled=1

priority=2

gpgcheck=1

type=rpm-md

gpgkey=http://download.ceph.com/keys/autobuild.asc

[apache2-ceph-source]

name=Apache source packages for Ceph

baseurl=http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/master/SRPMS

enabled=0

priority=2

gpgcheck=1

type=rpm-md

gpgkey=http://download.ceph.com/keys/autobuild.asc

5.安装ceph-deploy工具

该工具是python写的一个官方的ceph集群安装工具。

[root@vm-10-112-178-135 gadmin]#sudo yum update && sudo yum install ceph-deploy

6.查看是否安装了NTP服务，用来校准时间，因为每台机器通信，保证时间统一

7.安装openssh-server，配置ssh免密码连接。注意是在所有节点上。

在所有的节点上安装openssh-server

[root@vm-10-112-178-135 gadmin]# sudo yum install openssh-server

[root@vm-10-112-178-135 gadmin]#

添加用户名

[root@vm-10-112-178-135 gadmin]# sudo useradd -d /home/cephadmin -m cephadmin

[root@vm-10-112-178-135 gadmin]# passwd cephadmin

设置用户拥有root权限的免密码权限

[root@vm-10-112-178-135 gadmin]# echo "cephadmin ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadmin

cephadmin ALL = (root) NOPASSWD:ALL

[root@vm-10-112-178-135 gadmin]# sudo chmod 0440 /etc/sudoers.d/cephadmin

[cephadmin@vm-10-112-178-135 ~]$ ssh-keygen

[cephadmin@vm-10-112-178-135 .ssh]$ cd ~/.ssh/

[cephadmin@vm-10-112-178-135 .ssh]$ ll

总用量 8

-rw------- 1 cephadmin cephadmin 1679 7月  5 20:19 id_rsa

-rw-r--r-- 1 cephadmin cephadmin  409 7月  5 20:19 id_rsa.pub

[cephadmin@vm-10-112-178-135 .ssh]$ vi config

Copy the key to each Ceph Node, replacing {username} with the user name you created with Create a Ceph Deploy User.

ssh-copy-id {username}@node1

ssh-copy-id {username}@node2

ssh-copy-id {username}@node3

(Recommended) Modify the ~/.ssh/config file of your ceph-deploy admin node so that ceph-deploy can log in to Ceph nodes as the user you created without requiring you to specify --username {username} each time you execute ceph-deploy. This has the added benefit of streamlining ssh and scp usage. Replace {username} with the user name you created:

Host node1

Hostname node1

User {username}

Host node2

Hostname node2

User {username}

Host node3

Hostname node3

User {username}

Enable Networking On Bootup

Ceph OSDs peer with each other and report to Ceph Monitors over the network. If networking is off by default, the Ceph cluster cannot come online during bootup until you enable networking.

The default configuration on some distributions (e.g., CentOS) has the networking interface(s) off by default. Ensure that, during boot up, your network interface(s) turn(s) on so that your Ceph daemons can communicate over the network. For example, on Red Hat and CentOS, navigate to /etc/sysconfig/network-scripts and ensure that the ifcfg-{iface} file has ONBOOT set to yes.

[cephadmin@vm-10-112-178-135 my-cluster]$ cd ~/.ssh/

[cephadmin@vm-10-112-178-135 .ssh]$ ll

总用量 12

-rw-rw-r-- 1 cephadmin cephadmin  171 7月  5 20:26 config

-rw------- 1 cephadmin cephadmin 1679 7月  5 20:19 id_rsa

-rw-r--r-- 1 cephadmin cephadmin  409 7月  5 20:19 id_rsa.pub

[cephadmin@vm-10-112-178-135 .ssh]$ vi config

[cephadmin@vm-10-112-178-135 .ssh]$ cat config

Host 10.112.178.141

Hostname vm-10-112-178-141

User cephadmin

Host 10.112.178.142

Hostname vm-10-112-178-142

User cephadmin

Host 10.112.178.143

Hostname vm-10-112-178-143

User cephadmin

[cephadmin@vm-10-112-178-135 .ssh]$

[cephadmin@vm-10-112-178-135 .ssh]$ sudo ssh-copy-id -i ~/.ssh/id_rsa.pub  cephadmin@vm-10-112-178-141

The authenticity of host 'vm-10-112-178-141 (10.112.178.141)' can't be established.

ECDSA key fingerprint is af:7a:19:1d:76:b7:9b:51:0c:88:3a:4c:33:d0:f8:a5.

Are you sure you want to continue connecting (yes/no)? yes

/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

cephadmin@vm-10-112-178-141's password:

Number of key(s) added: 1

Now try logging into the machine, with:  "ssh 'cephadmin@vm-10-112-178-141'"

and check to make sure that only the key(s) you wanted were added.

[cephadmin@vm-10-112-178-135 .ssh]$ sudo ssh-copy-id -i ~/.ssh/id_rsa.pub  cephadmin@vm-10-112-178-142

The authenticity of host 'vm-10-112-178-142 (10.112.178.142)' can't be established.

ECDSA key fingerprint is af:7a:19:1d:76:b7:9b:51:0c:88:3a:4c:33:d0:f8:a5.

Are you sure you want to continue connecting (yes/no)? yes

/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

cephadmin@vm-10-112-178-142's password:

Number of key(s) added: 1

Now try logging into the machine, with:  "ssh 'cephadmin@vm-10-112-178-142'"

and check to make sure that only the key(s) you wanted were added.

[cephadmin@vm-10-112-178-135 .ssh]$ sudo ssh-copy-id -i ~/.ssh/id_rsa.pub  cephadmin@vm-10-112-178-143

The authenticity of host 'vm-10-112-178-143 (10.112.178.143)' can't be established.

ECDSA key fingerprint is af:7a:19:1d:76:b7:9b:51:0c:88:3a:4c:33:d0:f8:a5.

Are you sure you want to continue connecting (yes/no)? yes

/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

cephadmin@vm-10-112-178-143's password:

Number of key(s) added: 1

Now try logging into the machine, with:  "ssh 'cephadmin@vm-10-112-178-143'"

and check to make sure that only the key(s) you wanted were added.

[cephadmin@vm-10-112-178-135 .ssh]$ ssh cephadmin@10.112.178.141

Bad owner or permissions on /home/cephadmin/.ssh/config

[cephadmin@vm-10-112-178-135 .ssh]$ sudo firewall-cmd --zone=public --add-service=ceph-mon --permanent

FirewallD is not running

说明防火墙已经关闭了，这样就不用考率防火墙了。

[cephadmin@vm-10-112-178-135 ~]$ mkdir my-cluster

[cephadmin@vm-10-112-178-135 ~]$ cd my-cluster/

[cephadmin@vm-10-112-178-135 my-cluster]$

[cephadmin@vm-10-112-178-135 my-cluster]$ ceph-deploy new vm-10-112-178-135

[vm-10-112-178-135][DEBUG ] 总下载量：59 M[vm-10-112-178-135][DEBUG ] 安装大小：218 M[vm-10-112-178-135][DEBUG ] Downloading packages:[vm-10-112-178-135][WARNIN] No data was received after 300 seconds, disconnecting...[vm-10-112-178-135][INFO  ] Running command: sudo ceph --version[vm-10-112-178-135][ERROR ] Traceback (most recent call last):[vm-10-112-178-135][ERROR ]  File "/usr/lib/python2.7/site-packages/ceph_deploy/lib/vendor/remoto/process.py", line 119, in run[vm-10-112-178-135][ERROR ]    reporting(conn, result, timeout)[vm-10-112-178-135][ERROR ]  File "/usr/lib/python2.7/site-packages/ceph_deploy/lib/vendor/remoto/log.py", line 13, in reporting[vm-10-112-178-135][ERROR ]    received = result.receive(timeout)[vm-10-112-178-135][ERROR ]  File "/usr/lib/python2.7/site-packages/ceph_deploy/lib/vendor/remoto/lib/vendor/execnet/gateway_base.py", line 704, in receive[vm-10-112-178-135][ERROR ]    raise self._getremoteerror() or EOFError()[vm-10-112-178-135][ERROR ] RemoteError: Traceback (most recent call last):[vm-10-112-178-135][ERROR ]  File "/usr/lib/python2.7/site-packages/ceph_deploy/lib/vendor/remoto/lib/vendor/execnet/gateway_base.py", line 1036, in executetask[vm-10-112-178-135][ERROR ]    function(channel, **kwargs)[vm-10-112-178-135][ERROR ]  File "", line 12, in _remote_run

[vm-10-112-178-135][ERROR ]  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__

[vm-10-112-178-135][ERROR ]    errread, errwrite)

[vm-10-112-178-135][ERROR ]  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child

[vm-10-112-178-135][ERROR ]    raise child_exception

[vm-10-112-178-135][ERROR ] OSError: [Errno 2] No such file or directory

[vm-10-112-178-135][ERROR ]

[vm-10-112-178-135][ERROR ]

[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph --version

原因是网络比较慢,达到5分钟超时

解决方案:1.可以在每个节点上先行安装sudo yum -y install ceph

2.数量比较多的话多执行几次此命令

3.最佳方案是搭建本地源

[cephadmin@vm-10-112-178-135 my-cluster]$ sudo yum install -y ceph

[cephadmin@vm-10-112-178-135 my-cluster]$ sudo yum -y install ceph-radosgw

[ceph_deploy.cli][INFO  ]  dev                          : master

[ceph_deploy.cli][INFO  ]  nogpgcheck                    : False

[ceph_deploy.cli][INFO  ]  local_mirror                  : None

[ceph_deploy.cli][INFO  ]  release                      : None

[ceph_deploy.cli][INFO  ]  install_mon                  : False

[ceph_deploy.cli][INFO  ]  gpg_url                      : None

[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts vm-10-112-178-141

[ceph_deploy.install][DEBUG ] Detecting platform for host vm-10-112-178-141 ...

[vm-10-112-178-141][DEBUG ] connection detected need for sudo

sudo：抱歉，您必须拥有一个终端来执行 sudo

[ceph_deploy][ERROR ] RuntimeError: connecting to host: vm-10-112-178-141 resulted in errors: IOError cannot send (already closed?)

[cephadmin@vm-10-112-178-135 my-cluster]$

解决方案：

[摘要] Linux ssh执行远端服务器sudo命令时有如下报错：

sudo: sorry, you must have a tty to run sudo

sudo：抱歉，您必须拥有一个终端来执行 sudo

网上搜了一下，解决办法是编辑 /etc/sudoers 文件，将Default requiretty注释掉。

sudo vi /etc/sudoers

#Default requiretty #注释掉 Default requiretty 一行

具体操作：

sudo sed -i 's/Defaults    requiretty/#Defaults    requiretty/g' /etc/sudoers

sudo cat /etc/sudoers | grep requiretty

-----由于http代理网速太慢，只能选择分别再每台虚拟机节点上 手动安装 ceph，安装指令如下----

sudo yum -y install ceph

sudo yum -y install ceph-radosgw

[cephadmin@vm-10-112-178-135 my-cluster]$ ceph-deploy install vm-10-112-178-135 vm-10-112-178-141 vm-10-112-178-142 vm-10-112-178-143

ceph-deploy install {ceph-node}[{ceph-node} ...] --local-mirror=/opt/ceph-repo --no-adjust-repos --release=jewel

[cephadmin@vm-10-112-178-135 my-cluster]$ ceph-deploy osd prepare vm-10-112-178-142:/var/local/osd0 vm-10-112-178-143:/var/local/osd1

[vm-10-112-178-142][WARNIN] 2017-07-06 16:19:33.928816 7f2940a11800 -1  ** ERROR: error creating empty object store in /var/local/osd0: (13) Permission denied

[vm-10-112-178-142][WARNIN]

[vm-10-112-178-142][ERROR ] RuntimeError: command returned non-zero exit status: 1

[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /var/local/osd0

解决方案：

因为机器上的 /var/local/osd0 与 /var/local/osd1 这两个文件夹没有权限，使用 chmod 777 对文件夹 进行赋予 权限。

[cephadmin@vm-10-112-178-135 my-cluster]$ ceph-deploy osd activate vm-10-112-178-142:/var/local/osd0 vm-10-112-178-143:/var/local/osd1

最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 158,560评论 4赞 361
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 67,104评论 1赞 291
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 108,297评论 0赞 243
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 43,869评论 0赞 204
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 52,275评论 3赞 287
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 40,563评论 1赞 216
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 31,833评论 2赞 312
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 30,543评论 0赞 197
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 34,245评论 1赞 241
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 30,512评论 2赞 244
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 32,011评论 1赞 258
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 28,359评论 2赞 253
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 33,006评论 3赞 235
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 26,062评论 0赞 8
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 26,825评论 0赞 194
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 35,590评论 2赞 273
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 35,501评论 2赞 268

推荐阅读更多精彩内容

  • Ceph安装配置
    系统环境: centos73.10.0-514.26.2.el7.x86_64 机器数量：五台 硬盘：四块一块为系...
    think_lonely阅读 4,477评论 0赞 5
  • libvirt与ceph
    libvirt三种接口： 命令行：virsh 图形化：virt-manager Web：webvirtmgr 命令...
    Arteezy_Xie阅读 3,686评论 0赞 6
  • Linux搭建GlusterFS总结-详细流程步骤
    近期在linux上搭建了用于分布式存储的----GlusterFS和Ceph这两个开源的分布式文件系统。 前言--...
    ZNB_天玄阅读 4,889评论 0赞 7
  • ceph工作原理及安装
    一、概述 Ceph是一个分布式存储系统，诞生于2004年，最早致力于开发下一代高性能分布式文件系统的项目。随着云计...
    魏镇坪阅读 49,294评论 3赞 54
  • 如梦令
    《如梦令》 残梦更催秋瑟, 无奈泪湿衫袖。 挥别亦无语， 唯有眉黛微皱！ 此后 此后 应是南愁北瘦！
    刀剑啸九天阅读 401评论 1赞 2