Rook & Ceph 分布式存储

什么是 Rook

Rook 是 Kubernetes 的云原生存储编排器，将 Ceph 等分布式存储系统部署和管理为 K8s Operator。

安装 Rook-Ceph

# 克隆 Rook 仓库
git clone --single-branch --branch v1.13.0 https://github.com/rook/rook.git
cd rook/deploy/examples

# 安装 CRD 和 Operator
kubectl create -f crds.yaml -f common.yaml -f operator.yaml

# 创建 Ceph 集群
kubectl create -f cluster.yaml

Ceph 集群配置

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18.2.0
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 2
  dashboard:
    enabled: true
    ssl: true
  storage:
    useAllNodes: true
    useAllDevices: false
    devices:
    - name: sdb  # 使用 /dev/sdb 作为 OSD
    - name: sdc
  resources:
    osd:
      requests:
        cpu: "500m"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"

存储类型

RBD（块存储，RWO）

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
reclaimPolicy: Delete
allowVolumeExpansion: true

CephFS（文件存储，RWX）

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
  - name: replicated
    replicated:
      size: 3
  metadataServer:
    activeCount: 1
    activeStandby: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: myfs
  pool: myfs-replicated
reclaimPolicy: Delete
allowVolumeExpansion: true

常用操作

# 查看集群状态
kubectl exec -n rook-ceph deploy/rook-ceph-tools -- ceph status
kubectl exec -n rook-ceph deploy/rook-ceph-tools -- ceph osd status
kubectl exec -n rook-ceph deploy/rook-ceph-tools -- ceph df

# 访问 Ceph Dashboard
kubectl port-forward -n rook-ceph svc/rook-ceph-mgr-dashboard 8443:8443