[Ceph] Ceph Installation - jewel (latest 10.2.11) - II > Ceph 자료실

• 목록

안녕하세요.

 

꿈꾸는여행자입니다.

 

 

이번에는 Ceph을 주제로 다루고자 합니다. 

기존 Ceph 내용 구성 후 최근 Ceph 내용으로 검증해 나가 보겠습니다. 

> 다음 

 

1. Preflight

1.1. CEPH DEPLOY SETUP

1.2. CEPH NODE SETUP

                        

 

목차 

 

2. STORAGE CLUSTER QUICK START

2.1. Create a Deploy Folder

2.2. CREATE A CLUSTER

2.2.1. Purge the configuration [OPTIONS]

2.2.2. CREATE A CLUSTER

2.2.3. MODIFY THE CEPH CONFIGURATION FILE

2.2.4. INSTALL CEPH

2.2.5. ADD INITIAL MONITORS

2.2.6. ADJUST CRUSH TUNABLES

2.2.7. ADD OSDS

2.2.7.1. Delete the partition tables

2.2.7.2. Create two partition on the disk

2.2.7.2.1. Use an entire disk

2.2.7.2.2. Use a directory [OPTION]

2.2.7.3. Activate the OSDs

2.2.7.3.1. Use an entire disk

2.2.7.3.2. Use a directory [OPTION]

2.2.8. CRUSH HIERARCHY

2.2.8.1. CRUSH HIERARCHY

2.2.8.2. CREATE A CRUSH HIERARCHY

2.2.8.3. ADD OSD HOSTS/CHASSIS TO THE CRUSH HIERARCHY

2.2.8.4. CHECK CRUSH HIERARCHY

2.2.9. Copy ceph-deploy key

 

상세 내역은 아래와 같습니다.

 

감사합니다.  

 

> 아래 

 

________________

2. STORAGE CLUSTER QUICK START

http://docs.ceph.com/docs/master/start/quick-ceph-deploy/

If you haven’t completed your Preflight Checklist, do that first. This Quick Start sets up a Ceph Storage Cluster using ceph-deploy on your admin node. Create a three Ceph Node cluster so you can explore Ceph functionality.

        Storage cluster 실행 전 preflight가 미리 진행되어야 하며, ceph-deploy 명령어는 admin node에서 실행

  

부팅 시 osd node booting 후 monitoring node booting 하는 것이 수월함

As a first exercise, create a Ceph Storage Cluster with one Ceph Monitor and two Ceph OSD Daemons. 

Once the cluster reaches a active + clean state, expand it by adding a third Ceph OSD Daemon, a Metadata Server and two more Ceph Monitors. For best results, create a directory on your admin node node for maintaining the configuration files and keys that ceph-deploy generates for your cluster.

* 확장할 경우 

   * OSD Node 추가, 2개의 Monitors 추가 Metadata Server 추가가 필요함

   * Metadata Server가 어떤 용도로 필요로 하는지 확인이 필요

2.1. Create a Deploy Folder

admin node에 폴더를 생성하여 key 정보를 별도로 관리

# su - cephdeploy

[cephdeploy@admin-node ~]$ mkdir my-cluster

[cephdeploy@admin-node my-cluster]$ cd my-cluster

The ceph-deploy utility will output files to the current directory. Ensure you are in this directory when executing ceph-deploy.

* Important 

   * Do not call ceph-deploy with sudo or run it as root if you are logged in as a different user, because it will not issue sudo commands needed on the remote host.

* Disable requiretty

   * On some distributions (e.g., CentOS), you may receive an error while trying to execute ceph-deploy commands. If requiretty is set by default, disable it by executing sudo visudo and locate the Defaults requiretty setting. Change it toDefaults:ceph !requiretty to ensure that ceph-deploy can connect using the cephuser and execute commands with sudo.

2.2. CREATE A CLUSTER

2.2.1. Purge the configuration [OPTIONS]

* ceph-deploy 실행 시 문제 발생 할 경우 configuraion 초기화

* 일반 적인 경우 실행 할 필요 없음

If at any point you run into trouble and you want to start over, execute the following to purge the configuration:

ceph-deploy purgedata {ceph-node} [{ceph-node}]

ceph-deploy forgetkeys

* purgedata가 안되는 경우 

   * To remove all data from /var/lib/ceph (but leave Ceph packages intact), execute the purgedata command.

   * [cephdeploy@node1 ~]$ sudo rm -rf /var/lib/ceph

[cephdeploy@node2 ~]$ sudo rm -rf /var/lib/ceph

[cephdeploy@node3 ~]$ sudo rm -rf /var/lib/ceph

   * 

To purge the Ceph packages too, you may also execute:

ceph-deploy purge {ceph-node} [{ceph-node}]

If you execute purge, you must re-install Ceph.

2.2.2. CREATE A CLUSTER

On your admin node from the directory you created for holding your configuration details, perform the following steps using ceph-deploy.

* Create the cluster.

   * ceph-deploy new {initial-monitor-node(s)}

   *    * For example:

      1. monitor node 초기화

[cephdeploy@admin-node my-cluster]$ pwd

/home/cephdeploy/my-cluster

[cephdeploy@admin-node my-cluster]$ ceph-deploy new node1

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephdeploy/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.34): /bin/ceph-deploy new node1

[ceph_deploy.cli][INFO  ] ceph-deploy options:

[ceph_deploy.cli][INFO  ]  username                      : None

[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x28e3de8>

[ceph_deploy.cli][INFO  ]  verbose                       : False

[ceph_deploy.cli][INFO  ]  overwrite_conf                : False

[ceph_deploy.cli][INFO  ]  quiet                         : False

[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x2950d88>

[ceph_deploy.cli][INFO  ]  cluster                       : ceph

[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True

[ceph_deploy.cli][INFO  ]  mon                           : ['node1']

[ceph_deploy.cli][INFO  ]  public_network                : None

[ceph_deploy.cli][INFO  ]  ceph_conf                     : None

[ceph_deploy.cli][INFO  ]  cluster_network               : None

[ceph_deploy.cli][INFO  ]  default_release               : False

[ceph_deploy.cli][INFO  ]  fsid                          : None

[ceph_deploy.new][DEBUG ] Creating new cluster named ceph

[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds

[node1][DEBUG ] connected to host: admin-node 

[node1][INFO  ] Running command: ssh -CT -o BatchMode=yes node1

[node1][DEBUG ] connection detected need for sudo

[node1][DEBUG ] connected to host: node1 

[node1][DEBUG ] detect platform information from remote host

[node1][DEBUG ] detect machine type

[node1][DEBUG ] find the location of an executable

[node1][INFO  ] Running command: sudo /usr/sbin/ip link show

[node1][INFO  ] Running command: sudo /usr/sbin/ip addr show

[node1][DEBUG ] IP addresses found: ['10.0.1.70', '10.0.2.15', '10.0.0.70']

[ceph_deploy.new][DEBUG ] Resolving host node1

[ceph_deploy.new][DEBUG ] Monitor node1 at 10.0.0.70

[ceph_deploy.new][DEBUG ] Monitor initial members are ['node1']

[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.0.0.70']

[ceph_deploy.new][DEBUG ] Creating a random mon key...

[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...

[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

* Check the output of ceph-deploy with ls and cat in the current directory. You should see a Ceph configuration file, a monitor secret keyring, and a log file for the new cluster. See ceph-deploy new -h for additional details.

2.2.3. MODIFY THE CEPH CONFIGURATION FILE

1. Change the default number of replicas in the Ceph configuration file from 3 to 2 so that Ceph can achieve an active + clean state with just two Ceph OSDs. Add the following line under the [global] section of ceph.conf file in my-cluster directory::

   * replica 숫자를 변경

[cephdeploy@admin-node my-cluster]$ vi /home/cephdeploy/my-cluster/ceph.conf

[global]

...

osd pool default size = 2

osd_pool_default_min_size = 2

   * 2. If you have more than one network interface, add the public network and cluster_network setting under the [global] section of your Ceph configuration file. See the Network Configuration Reference for details.

   * [cephdeploy@admin-node my-cluster]$ vi /home/cephdeploy/my-cluster/ceph.conf

public_network = <ip-address>/<netmask>

cluster_network = <ip-address>/<netmask>

[global]

...

public_network = 10.0.0.1/24

cluster_network = 10.0.1.1/24

   * 3. These settings distinguish which network is public (front-side) and which network is for the cluster (back-side). Ensure that your nodes have interfaces configured for these networks. We do not recommend using the same NIC for the public and cluster networks. Please see the Network Configuration Settings for details on the public and cluster networks.

4. Turn on IPv6 if you intend to use it. [OPTION]

   * ms_bind_ipv6 = true

   * 5. Add or adjust the osd journal size setting under the [global] section of your Ceph configuration file.

   * [cephdeploy@admin-node my-cluster]$ vi /home/cephdeploy/my-cluster/ceph.conf

[global]

...

osd_journal_size = 10000

   * Desc

      1. We recommend a general setting of 10GB. Ceph’s default osd_journal_size is 0, so you will need to set this in your ceph.conf file. A journal size should be the product of the filestore_max_sync_interval option and the expected throughput, and then multiply the resulting product by two. The expected throughput number should include the expected disk throughput (i.e., sustained data transfer rate), and network throughput. For example, a 7200 RPM disk will likely have approximately 100 MB/s. Taking the min() of the disk and network throughput should provide a reasonable expected throughput. Please see Journal Settings for more details.

6. Set a CRUSH leaf type to the largest serviceable failure domain for your replicas under the [global] section of your Ceph configuration file. The default value is 1, or host, which means that CRUSH will map replicas to OSDs on separate separate hosts. For example, if you want to make three object replicas, and you have three racks of chassis/hosts, you can set osd_crush_chooseleaf_type to 3, and CRUSH will place each copy of an object on OSDs in different racks.

   * [cephdeploy@admin-node my-cluster]$ vi /home/cephdeploy/my-cluster/ceph.conf

[global]

...

osd_crush_chooseleaf_type = 3

   * The default CRUSH hierarchy types are:

      1. type 0 osd

      2. type 1 host

      3. type 2 chassis

      4. type 3 rack

      5. type 4 row

      6. type 5 pdu

      7. type 6 pod

      8. type 7 room

      9. type 8 datacenter

      10. type 9 region

      11. type 10 root

7. Set max_open_files so that Ceph will set the maximum open file descriptors at the OS level to help prevent Ceph OSD Daemons from running out of file descriptors.

   * [cephdeploy@admin-node my-cluster]$ vi /home/cephdeploy/my-cluster/ceph.conf

[global]

...

max_open_files = 131072

   * 8. In summary, your initial Ceph configuration file should have at least the following settings with appropriate values assigned after the = sign:

   * [global]

fsid = <cluster-id>

mon_initial_members = <hostname>[, <hostname>]

mon_host = <ip-address>[, <ip-address>]

public_network = <network>[, <network>]

cluster_network = <network>[, <network>]

ms_bind_ipv6 = [true | false]

max_open_files = 131072

auth_cluster_required = cephx

auth_service_required = cephx

auth_client_required = cephx

osd_journal_size = <n>

filestore_xattr_use_omap = true

osd_pool_default_size = <n> # Write an object n times.

osd_pool_default_min_size = <n> # Allow writing n copy in a degraded

state.

osd_crush_chooseleaf_type = <n>

[global]

fsid = 88135ade-3647-4083-bcc7-b4d7a64d2849

mon_initial_members = node1

mon_host = 10.0.0.70

auth_cluster_required = cephx

auth_service_required = cephx

auth_client_required = cephx

osd pool default size = 2

osd_pool_default_min_size = 2

public_network = 10.0.0.1/24

cluster_network = 10.0.1.1/24

osd_journal_size = 10000

osd_crush_chooseleaf_type = 3

max_open_files = 131072

   * 2.2.4. INSTALL CEPH

[admin-node에서 cephdeploy 계정으로 my-cluster 디렉토리에서 실행]

1. Install Ceph.

   * ceph-deploy install {ceph-node}[{ceph-node} ...]

   *    * For example:

      1. [cephdeploy@admin-node my-cluster]$ pwd

/home/cephdeploy/my-cluster

[cephdeploy@admin-node my-cluster]$ ceph-deploy install admin-node node1 node2 node3

      2.    * The ceph-deploy utility will install Ceph on each node. NOTE: If you use ceph-deploy purge, you must re-execute this step to re-install Ceph.

2. Error Case : ceph repo

   * [admin-node][WARNIN] check_obsoletes has been enabled for Yum priorities plugin

[admin-node][INFO  ] Running command: sudo rpm --import https://download.ceph.com/keys/release.asc

[admin-node][INFO  ] Running command: sudo rpm -Uvh --replacepkgs https://download.ceph.com/rpm-jewel/el7/noarch/ceph-release-1-0.el7.noarch.rpm

[admin-node][DEBUG ] https://download.ceph.com/rpm-jewel/el7/noarch/ceph-release-1-0.el7.noarch.rpm(을)를 복구합니다

[admin-node][DEBUG ] Updating / installing...

[admin-node][WARNIN] 경고: /etc/yum.repos.d/ceph.repo(이)가 /etc/yum.repos.d/ceph.repo.rpmnew(으)로 생성되었습니다

[ceph_deploy][ERROR ] RuntimeError: NoSectionError: No section: 'ceph'

   *    * Solve

      1. 기존 repo정보를 임시로 이동 시킨 후 ceph.repo

# cd /etc/yum.repos.d/

# sudo mv ceph.repo ~/

# sudo mv ceph.repo.rpmnew ceph.repo

      2. 2.2.5. ADD INITIAL MONITORS

1. Add the initial monitor(s) and gather the keys:

   * monitor node 초기화

[cephdeploy@admin-node my-cluster]$ ceph-deploy mon create-initial

   *    * Once you complete the process, your local directory should have the following keyrings:

   * 완료 후 키 정보 확인

      1. {cluster-name}.client.admin.keyring

      2. {cluster-name}.bootstrap-osd.keyring

      3. {cluster-name}.bootstrap-mds.keyring

      4. {cluster-name}.bootstrap-rgw.keyring

   * Note 

      1. The bootstrap-rgw keyring is only created during installation of clusters running Hammer or newer

      2. If this process fails with a message similar to “Unable to find /etc/ceph/ceph.client.admin.keyring”, please ensure that the IP listed for the monitor node in ceph.conf is the Public IP, not the Private IP.

   * 2. Copy keyring file and ceph.conf

   * [cephdeploy@admin-node my-cluster]$ pwd

/home/cephdeploy/my-cluster

[cephdeploy@admin-node my-cluster]$ sudo cp *.keyring /etc/ceph/

[cephdeploy@admin-node my-cluster]$ sudo cp ceph.conf /etc/ceph/

[cephdeploy@admin-node my-cluster]$ sudo chmod 644 /etc/ceph/*.keyring  /etc/ceph/*.conf

   * 2.2.6. ADJUST CRUSH TUNABLES

Ceph Storage CRUSH tunables defaults to bobtail, which refers to an older release of Ceph. This setting guarantees that older Ceph clusters are compatible with older Linux kernels. However, if you run a Ceph cluster on Red Hat Enterprise Linux 7, reset CRUSH tunables to optimal: 

[cephdeploy@admin-node ceph]$ ceph osd crush tunables optimal

adjusted tunables profile to optimal

2.2.7. ADD OSDS

* Before creating OSDs, consider the following: 

   * We recommend using the XFS file system, which is the default file system.

   * Warning 

      * Use the default XFS file system options that the ceph-deploy utility uses to format the OSD disks. Deviating from the default values can cause stability problems with the storage cluster. For example, setting the directory block size higher than the default value of 4096 bytes can cause memory allocation deadlock errors in the file system. For more details, view the Red Hat Knowledgebase article regarding these errors.

   * Red Hat recommends using SSDs for journals. It is common to partition SSDs to serve multiple OSDs. Ensure that the number of SSD partitions does not exceed the SSD’s sequential write limits. Also, ensure that SSD partitions are properly aligned, or their write performance will suffer. 

2.2.7.1. Delete the partition tables

* Red Hat recommends to delete the partition table of a Ceph OSD drive by using the ceph-deploy disk zap command before executing the ceph-deploy osd prepare command:

   * disk zap 명령어로 partition table 삭제

$ ceph-deploy disk zap <ceph_node>:<disk_device>

   * For example: 

      * [cephdeploy@admin-node my-cluster]$ ceph-deploy disk zap node2:sdb

[cephdeploy@admin-node my-cluster]$ ceph-deploy disk zap node3:sdb

      * 

2.2.7.2. Create two partition on the disk

2.2.7.2.1. Use an entire disk

* The prepare command creates two partitions on a disk device; one partition is for OSD data, and the other is for the journal.

   * prepare 명령어로 해당 테이블 파티션 생성

* From your administration node, use ceph-deploy osd prepare to prepare the OSDs:

   * $ ceph-deploy osd prepare <ceph_node>:<disk_device> [<ceph_node>:<disk_device>]

   * For example: 

      * [cephdeploy@admin-node my-cluster]$ ceph-deploy osd prepare node2:/dev/sdb

[cephdeploy@admin-node my-cluster]$ ceph-deploy osd prepare node3:/dev/sdb

      * * prepare 후 해당 disk 정보

   * [root@node2 ~]# fdisk /dev/sdd

WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Command (m for help): p

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk label type: gpt

#         Start          End    Size  Type            Name

 1     20482048   1953525134  921.8G  unknown         ceph data

 2         2048     20482047    9.8G  unknown         ceph journal

   * 2.2.7.2.2. Use a directory [OPTION]

* For use a directory

   * 디렉토리로 사용하여 하는 경우

   * Add two OSDs. For fast setup, this quick start uses a directory rather than an entire disk per Ceph OSD Daemon. See ceph-deploy osd for details on using separate disks/partitions for OSDs and journals. Login to the Ceph Nodes and create a directory for the Ceph OSD Daemon.

   * [root@admin-node ~]# su - cephdeploy

[cephdeploy@admin-node ~]$ ssh node2

[cephdeploy@node2 ~]$ sudo mkdir /var/local/osd0

[cephdeploy@node2 ~]$ sudo chown -R ceph:ceph /var/local/osd0

[cephdeploy@node2 ~]$ exit

[cephdeploy@admin-node ~]$  ssh node3

[cephdeploy@node3 ~]$ sudo mkdir /var/local/osd1

[cephdeploy@node3 ~]$ sudo chown -R ceph:ceph /var/local/osd1

[cephdeploy@node3 ~]$ exit

   * * Then, from your admin node, use ceph-deploy to prepare the OSDs.

   * [cephdeploy@admin-node ~]$ cd ~/my-cluster

[cephdeploy@admin-node my-cluster]$

 ceph-deploy osd prepare node2:/var/local/osd0 node3:/var/local/osd1

   * 

2.2.7.3. Activate the OSDs

2.2.7.3.1. Use an entire disk

* Once you prepare OSDs, activate the OSDs:

   * $ ceph-deploy osd activate <ceph_node>:<data_partition>

   *    * For example:

      * [cephdeploy@admin-node my-cluster]$ ceph-deploy osd activate node2:/dev/sdb1

[cephdeploy@admin-node my-cluster]$ ceph-deploy osd activate node3:/dev/sdb1

      *       * Note

         * In the ceph-deploy osd activate command, specify a particular disk partition, for example /dev/sdb1.

         * activate 명령어 사용 시는 특정 partition 선택이 필요

2.2.7.3.2. Use a directory [OPTION]

* Finally, activate the OSDs.

   * [cephdeploy@admin-node my-cluster]$ pwd

/home/cephdeploy/my-cluster

[cephdeploy@admin-node my-cluster]$ ceph-deploy osd activate node2:/var/local/osd0 node3:/var/local/osd1

   * 

2.2.8. CRUSH HIERARCHY

* CRUSH MAPS

   * http://docs.ceph.com/docs/master/rados/operations/crush-map/

* CRUSH ADMINISTRATION

   * https://access.redhat.com/documentation/en/red-hat-ceph-storage/version-1.3/storage-strategies-guide/#crush_administration

2.2.8.1. CRUSH HIERARCHY

만들고자 하는 crush hierarchy

  

변경전 osd tree 확인

[cephdeploy@admin-node my-cluster]$ ceph osd tree

ID WEIGHT  TYPE NAME      UP/DOWN REWEIGHT PRIMARY-AFFINITY 

-1 0.09760 root default                                     

-2 0.04880     host node2                                   

 1 0.04880         osd.1       up  1.00000          1.00000 

-3 0.04880     host node3                                   

 2 0.04880         osd.2       up  1.00000          1.00000 

 0       0 osd.0             down        0          1.00000 

2.2.8.2. CREATE A CRUSH HIERARCHY

You can run a Ceph cluster with a flat node-level hierarchy (default). This is NOT RECOMMENDED. We recommend adding named buckets of various types to your default CRUSH hierarchy. This will allow you to establish a larger-grained failure domain, usually consisting of racks, rows, rooms and data centers.

ceph osd crush add-bucket <bucket-name> <bucket-type>

For example:

ceph osd crush add-bucket dc1 datacenter

ceph osd crush add-bucket room1 room

ceph osd crush add-bucket row1 row

ceph osd crush add-bucket rack1 rack

ceph osd crush add-bucket rack2 rack

ceph osd crush add-bucket rack3 rack

To move a bucket to a different location or position in the CRUSH map hierarchy, execute the following:

ceph osd crush move {bucket-name} {bucket-type}={bucket-name}, [...]

Then, place the buckets into a hierarchy:

ceph osd crush move dc1 root=default

ceph osd crush move room1 datacenter=dc1

ceph osd crush move row1 room=room1

ceph osd crush move rack1 row=row1

ceph osd crush move rack2 row=row1

ceph osd crush move rack3 row=row1

ceph osd crush move node2 rack=rack1

ceph osd crush move node3 rack=rack2

* Desc

   * root / datacetner / room / row / rack 형태로 트리 생성

   * rack에 각각의 rack1, rack2, rack3를 두어 replication 시 정상적으로 동작하게 설정

tree 확인

[cephdeploy@admin-node my-cluster]$ ceph osd tree

ID WEIGHT  TYPE NAME                      UP/DOWN REWEIGHT PRIMARY-AFFINITY 

-1 2.00000 root default                                                     

-4 2.00000     datacenter dc1                                               

-5 2.00000         room room1                                               

-6 2.00000             row row1                                             

-7 1.00000                 rack rack1                                       

-2 0.00000                     host node2                                   

 1 1.00000                         osd.1       up  1.00000          1.00000 

-8 1.00000                 rack rack2                                       

-3 1.00000                     host node3                                   

 2 1.00000                         osd.2       up  1.00000          1.00000 

-9       0                 rack rack3                                       

 0       0 osd.0                             down        0          1.00000