Ceph Clusters

Production Clusters
Cluster Lead Use-case Mon host (where?) Release Version OS Racks IP Services Power SSB Upgrades?
barn Enrico Cinder:
cp1, cpio1 		cephbarn (hw) pacific 16.2.9-1 RHEL8 BA09 S513-A-IP250 UPS-4/-C Yes
beesly Enrico Glance
Cinder:
1st AZ 		cephmon (hw) pacific 16.2.9-1 RHEL8 CD27-CD30 S513-C-IP152 UPS-3/-4i Yes
cta Roberto CTA prod cephcta (hw) pacific 16.2.13-5 RHEL8 SI36-SI41 - No, Julien Leduc
dwight Zac Testing + Manila: CephFS Testing cephmond (vm,abc) quincy 17.2.7-1 Alma8 CE01-CE03 S513-C-IP501 Yes + Manila MM
doyle CephFS for DFS Projects cephdoyle (hw) quincy 17.2.7-2 RHEL9 CP18, CP19-21, CP22 S513-C-IP200 UPS-1 Yes + Sebast/Giuseppe
flax(*) Abhi Manila:
Meyrin CephFS 		cephflax (vm,abc) pacific 16.2.9-1 RHEL8 BA10,SQ05
CQ18-CQ21
SJ04-SJ07 		S513-A-IP558,S513-V-IP562
S513-C-IP164
S513-V-IP553 		UPS-4/-C,UPS-1
UPS-1
UPS-3 		Yes
gabe Enrico S3 cephgabe (hw) pacific 16.2.13-5 RHEL8 SE04-SE07
SJ04-SJ07 		S513-V-IP808
S513-V-IP553 		UPS-1
UPS-3 		Yes
jim Enrico HPC BE (CephFS) cephjim (vm,abc) quincy 17.2.7-1 RHEL8 SW11-SW15
SX11-SX15 		S513-V-IP194
S513-V-IP193 		UPS-3
UPS-3 		Yes + Nils Hoimyr
kelly Roberto Cinder: hyperc +
CTA preprod 		cephkelly (hyperc) quincy 17.2.7-1 RHEL8 CQ12-CQ22 S513-C-IP164 UPS-1 Yes + Julien Leduc
kapoor Enrico Cinder:
cpio2, cpio3 		cephkapoor (hyperc) quincy 17.2.7-1 RHEL8 BE10 BE11 BE13 S513-A-IP22 UPS-4/-C Yes
levinson Abhi Manila:
Meyrin CephFS SSD A 		cephlevinson (hw) pacific 16.2.9-1 RHEL8 BA03 BA04 BA05 BA07 S513-A-IP120 S513-A-IP119 S513-A-IP121 S513-A-IP122 UPS-4/-C Yes
meredith Enrico Cinder: io2, io3 cephmeredith (hw) pacific 16.2.9-1 RHEL8 CK01-23 S513-C-IP562 UPS-2 Yes
nethub Enrico S3 FR + Cinder FR cephnethub (hw) pacific 16.2.13-5 RHEL8 HA06-HA09
HB01-HB06 		S773-C-SI180
S773-C-IP200 		EOD104,ESK404
EOD105 (CEPH-1519) 		Yes
pam Abhi Manila:
Meyrin CephFS B 		cephpam (hw) pacific 16.2.9-1 Alma8 CP16-19 S513-C-IP200 UPS-1 Yes
poc Enrico PCC Proof of Concept (CEPH-1382) cephpoc (hyperc) quincy 17.2.7-1 RHEL9 SU06 S513-V-SI263 No
ryan Enrico Cinder: 3rd AZ cephryan (hw) pacific 16.2.9-1 RHEL8 CE01-CE03 S513-C-IP501 UPS-2 Yes
stanmey Zachary S3 multi-site, Meyrin (secondary) cephstanmey (hw) reef 18.2.1-1 RHEL8 CP16-24 S513-C-IP200 UPS-1 No
stanpre Zachary S3 multi-site, Prevessin (master) cephstanpre (hw) reef 18.2.1-1 Alma8 HB01-HB06 S773-C-IP200 EOD105/0E No
toby Enrico Stretch cluster cephtoby (hw) pacific 16.2.9-1 RHEL8 CP16-19
SJ04-07 		S513-C-IP200
S513-V-IP553 		UPS-1
UPS-3 		No
vance Enrico Manila: HPC Theory-QCD cephvance (hw) quincy 17.2.7-1 Alma8 CP16-CP17, CP19, CP21, CP23-CP24 S513-C-IP200 UPS-1 Yes + Nils Hoimyr
wallace Enrico krbd: Oracle DB restore tests cephwallace (hw) quincy 17.2.7-1 RHEL8 CP18, CP20, CP22 S513-C-IP200 UPS-1 No, Dmytro Grzudo
vault Enrico Cinder: 2nd AZ cephvault (hw) pacific 16.2.9-1 RHEL8 SE04-SE07 S513-V-IP808 UPS-1 Yes

Flax locations details:

  • MONs: 3x OpenStack VMs, one in each availability zone
  • MDSes (CPU servers): 50% in barn, 50% in vault
    • cephcpu21-0c370531cf, SQ05, S513-V-IP562, UPS 1 (EOD1*43)
    • cephcpu21-2456968853, SQ05, S513-V-IP562, UPS 1 (EOD1*43)
    • cephcpu21-46bb400fc8, BA10, S513-A-IP558
    • cephcpu21-4a93514bf3, BA10, S513-A-IP558
    • cephcpu21b-417b05bfee, BA10, S513-A-IP558
    • cephcpu21b-4ad1d0ae5f, SQ05, S513-V-IP562, UPS 1 (EOD1*43)
    • cephcpu21b-a703fac16c, SQ05, S513-V-IP562, UPS 1 (EOD1*43)
    • cephcpu21b-aecbee75a5, BA10, S513-A-IP558
  • Metadata pool: Main room, UPS-1 EOD1*43
  • Data pool: Vault, UPS-3 EOD3*43

Each production cluster has a designated cluster lead, who is the primary contact and responsible for that cluster.

The user-visible "services" provided by the clusters are documented in our Service Availability probe: https://gitlab.cern.ch/ai/it-puppet-hostgroup-ceph/-/blob/qa/code/files/sls/ceph-availability-producer.py#L19

The QoS provided by each user-visible cluster is described in OpenStack docs. Cinder volumes available on multiple AZs are of standard and io1 types.

s3.cern.ch RGWs
Hostname Customer IPv4 IPv6 IPsvc VM IPsvc Real Runs on OpenStack AZ Room Rack Power
cephgabe-rgwxl-325de0fb1d cvmfs 137.138.152.241 2001:1458:d00:13::1e5 S513-C-VM33 0513-C-IP33 P06636663U66968 cern-geneva-a main CH14 UPS-3
cephgabe-rgwxl-86d4c90cc6 cvmfs 137.138.33.24 2001:1458:d00:18::390 S513-V-VM936 0513-V-IP35 P06636688Q51842 cern-geneva-b vault SQ27 UPS-4
cephgabe-rgwxl-8930fc00f8 cvmfs 137.138.151.203 2001:1458:d00:12::3e0 S513-C-VM32 0513-C-IP32 P06636663N63480 cern-geneva-c main CH11 UPS-3
cephgabe-rgwxl-8ee4a698b7 cvmfs 137.138.44.245 2001:1458:d00:1a::24b S513-C-VM933 0513-C-IP33 P06636663J50924 cern-geneva-a main CH16 UPS-3
cephgabe-rgwxl-3e0d67a086 default 188.184.73.131 2001:1458:d00:4e::100:4ae S513-A-VM805 0513-A-IP561 I82006520073152 cern-geneva-c barn BC11 UPS-4/-C
cephgabe-rgwxl-652059ccf1 default 188.185.87.72 2001:1458:d00:3f::100:2bd S513-A-VM559 0513-A-IP559 I82006525008611 cern-geneva-a barn BC06 UPS-4/-C
cephgabe-rgwxl-8e7682cb81 default 137.138.158.145 2001:1458:d00:14::341 S513-V-VM35 0513-V-IP35 P06636688R71189 cern-geneva-b vault SQ28 UPS-4
cephgabe-rgwxl-91b6e0d6dd default 137.138.77.21 2001:1458:d00:1c::405 S513-C-VM931 0513-C-IP33 P06636663M67468 cern-geneva-a main CH13 UPS-3
cephgabe-rgwxl-895920ea1a gitlab 137.138.158.221 2001:1458:d00:14::299 S513-V-VM35 0513-V-IP35 P06636688H41037 cern-geneva-b vault SQ29 UPS-4
cephgabe-rgwxl-9e3981c77a gitlab 137.138.154.49 2001:1458:d00:13::3a S513-C-VM33 0513-C-IP33 P06636663J50924 cern-geneva-a main CH16 UPS-3
cephgabe-rgwxl-dbb0bcc513 gitlab 188.184.102.175 2001:1458:d00:3b::100:2a9 S513-C-VM852 0513-C-IP852 I78724428177369 cern-geneva-c main EK03 UPS-2
cephgabe-rgwxl-26774321ac jec-data 188.185.10.120 2001:1458:d00:63::100:39a S513-V-VM902 0513-V-IP402 I88681450454656 cern-geneva-a vault SP23 UPS-4
cephgabe-rgwxl-a273d35b9d jec-data 188.185.19.171 2001:1458:d00:65::100:32a S513-V-VM406 S513-V-IP406 I88681458914473 cern-geneva-b vault SP27 UPS-4
cephgabe-rgwxl-d91c221898 jec-data 137.138.155.51 2001:1458:d00:13::14d S513-C-VM33 0513-C-IP33 P06636663Y16806 cern-geneva-a main CH15 UPS-3
cephgabe-rgwxl-75569ebe5c prometheus 137.138.149.253 2001:1458:d00:12::52f S513-C-VM32 0513-C-IP32 P06636663G98563 cern-geneva-c main CH04 UPS-3
cephgabe-rgwxl-7658b46c78 prometheus 188.185.9.237 2001:1458:d00:63::100:424 S513-V-VM902 0513-V-IP402 I88681457779137 cern-geneva-a vault SP24 UPS-4
cephgabe-rgwxl-05386c6cdb vistar 188.185.86.117 2001:1458:d00:3f::100:2d9 S513-A-VM559 0513-A-IP559 I82006526449210 cern-geneva-a barn BC05 UPS-4/-C
cephgabe-rgwxl-13f36a01c2 vistar 137.138.33.10 2001:1458:d00:18::1ee S513-V-VM936 0513-V-IP35 P06636688C41209 cern-geneva-b vault SQ29 UPS-4
cephgabe-rgwxl-6da6da7653 vistar 188.184.74.136 2001:1458:d00:4e::100:5d S513-A-VM805 0513-A-IP561 I82006527765435 cern-geneva-c barn BC13 UPS-4/-C

Reviewing a Cluster Status
  1. Check Grafana dashboards for unusual activity, patterns, memory usage:
  • https://filer-carbon.cern.ch/grafana/d/000000001/ceph-dashboard
  • https://filer-carbon.cern.ch/grafana/d/000000108/ceph-osd-mempools
  • https://filer-carbon.cern.ch/grafana/d/uHevna1Mk/ceph-hosts
  • For RGWs: https://filer-carbon.cern.ch/grafana/d/iyLKxjoGk/s3-rgw-perf-dumps
  • For CephFS: * https://filer-carbon.cern.ch/grafana/d/000000111/cephfs-detail
  • etc...
  1. Login to cluster mon and check various things:
  • ceph osd pool ls detail - are the pool flags correct? e.g. nodelete,nopgchange,nosizechange
  • ceph df - assess amount of free space for capacity planning
  • ceph osd crush rule ls, ceph osd crush rule dump - are the crush rules as expected?
  • ceph balancer status - as expected?
  • ceph osd df tree - are the PGs per OSD balanced and a reasonable number, e.g. < 100.
  • ceph osd tree out, ceph osd tree down - are there any OSDs that are not being replaced properly?
  • ceph config dump - is the configuration as expected?
  • ceph telemetry status - check from config if it on, enable it

Clusters' priority

In case of a major incident (e.g., power cuts), revive clusters in the following order:

  1. Beesly (RBD1, main, UPS-3/4), Flax (CephFS, everywhere), Gabe (S3, vault, UPS-1/3)
  2. Vault (RBD2, vault, UPS-1), Levinson (CephFS SSD, vault, UPS-1), Meredith (RBD SSD, main, UPS-2)
  3. Ryan (RBD3, main, UPS-2), CTA (ObjectStore, vault, UPS-1)
  4. Jim, Dwight, Kelly, Pam (currently unused)
  5. Barn, Kopano -- should not go down, as they are in critical power
  6. NetHub -- 2nd network hub, Prevessin, diesel-backed (9/10 racks)

Hardware Specs

Test clusters
Cluster Use-case Mon alias Release Version Notes
cslab Test cluster for Network Lab (RQF2068297,CEPH-1348) cephcslab pacific 16.2.13-5 Binds to IPv6 only; 3 hosts Alma8 + 3 RHEL8
miniflax Mini cluster mimicking Flax None (ceph/miniflax/mon) quincy 17.2.7-1
minigabe Mini cluster mimicking Gabe (zone groups) cephminigabe pacific 16.2.13-6 RGW on minigabe-831ffcf9f9; Beast on 8080; RGW DNS: cephminigabe
next RC and Cloud next region testing cephnext01 quincy 17.2.6-4
Improve me !