kubernetes扩容存储卷预研.md


title: 'Resize' pinned: false tags: [Notebooks/Tutorial]


随着越来越来的容器开始跑有状态服务,因此容器存储的需求日渐增加 因此在这里预研容器扩容存储的场景 卷扩容功能现在还是beta状态,但是也可以在在csi driver中实现具体的接口进行实验性使用。

原理描述

正常创建PVC后发生了什么事情:

provision

  1. kubectl 创建了一个PVC对象, 1.5)API server收到该请求并对数据校验后写入了etcd中
  2. 一个外部的controller(external provisoner)会订阅PVC的事件,此时这个ADD事件引起他的注意
  3. provisioner通过storageclass找到相应的csi driver(存储商提供)并向driver 发起RPC请求CreateVolume接口
  4. CSI driver执行存储端相应的操作,吐出了一个PV对象 至此,PVC通过结合storageclass提供的driver完成PV的自动创建和绑定 后续的attach和mount操作此处略过

现在问题来了

假设现在PV已经吐出来,并在Pod中已经消费使用

使用了一段时间发现不够容量用,咋整?

满足以下条件:

  • csi 规范里有预留了扩容存储卷的功能,只是在1.0版本暂时还不使用

  • 可以开启feature gate进行先行体验

  • 需要底层存储支持扩容

如果相应的driver实现了扩容接口,且k8s集群开启扩容特性(笔者写这篇文章的时候该特性还没GA),那么这个时候就不用慌,只需对相应的PVC edit一下,就会触发PVC的更新。然后扩容就可以继续使用。 如下图所示: resizer

  1. kubectl edit PVC 1.5) edit后就触发PVC对象在etcd侧更新
  2. 此时另一个sidecar容器叫external resizer表示对PVC也很感兴趣,他也订阅了PVC的事件,因此知道了此次PVC 的Update事件
  3. External resizer被UPDATE事件触发,然后找到相应的CSI driver,并发起RPC请求ControllerExpandVolume
  4. 如果driver支持该操作,他就和后段存储相应扩容,然后吐出新的PV size

至此pod就可以使用新的存储容量了,(其实不完全准确,因为有的存储还需要在NodeExpandVolume接口中实现)

实操指南

开启功能

1) kube-controller-manager在feature gate开启该特性 vim /etc/kubernetes/manifests/kube-controller-manager.yaml featuregate1

2) 在api server开启该特性 vim /etc/kubernetes/manifests/kube-apiserver.yaml featuregate2

3) 重启kubelet重新加载配置

4) 添加rbac权限给辅助容器

新建个rbac.yaml,填入以下内容

# This YAML file contains all RBAC objects that are necessary to run external
# CSI resizer.
#
# In production, each CSI driver deployment has to be customized:
# - to avoid conflicts, use non-default namespace and different names
# for non-namespaced entities like the ClusterRole
# - decide whether the deployment replicates the external CSI
# resizer, in which case leadership election must be enabled;
# this influences the RBAC setup, see below

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-resizer
  # replace with non-default namespace name
  namespace: default

---
# Resizer must be able to work with PVCs, PVs, SCs.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-resizer-runner
rules:
  # The following rule should be uncommented for plugins that require secrets
  # for provisioning.
  # - apiGroups: [""]
  # resources: ["secrets"]
  # verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims/status"]
    verbs: ["update", "patch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-resizer-role
subjects:
  - kind: ServiceAccount
    name: csi-resizer
    # replace with non-default namespace name
    namespace: default
roleRef:
  kind: ClusterRole
  name: external-resizer-runner
  apiGroup: rbac.authorization.k8s.io

---
# Resizer must be able to work with end point in current namespace
# if (and only if) leadership election is enabled
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  # replace with non-default namespace name
  namespace: default
  name: external-resizer-cfg
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-resizer-role-cfg
  # replace with non-default namespace name
  namespace: default
subjects:
  - kind: ServiceAccount
    name: csi-resizer
    # replace with non-default namespace name
    namespace: default
roleRef:
  kind: Role
  name: external-resizer-cfg
  apiGroup: rbac.authorization.k8s.io

并创建kubectl create -f rbac.yaml

5) 创建相应的辅助容器resizer

该容器负责监听PVC的变化并触发扩容操作 新建deployment.yaml,填入下面内容 注意替换hostPath 和socket为你部署的driver

# This YAML file demonstrates how to deploy the external
# resizer for use with the mock CSI driver. It
# depends on the RBAC definitions from rbac.yaml.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: csi-resizer
spec:
  replicas: 1
  selector:
    matchLabels:
      external-resizer: mock-driver
  template:
    metadata:
      labels:
        external-resizer: mock-driver
    spec:
      serviceAccount: csi-resizer
      containers:
        - name: csi-resizer
          image: quay.io/k8scsi/csi-resizer:canary
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins_registry/csi-xsky-nfs-driver/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/
            - name: plugin-dir
              mountPath: /var/lib/kubelet/plugins_registry/csi-xsky-nfs-driver

      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/csi-xsky-nfs-driver
            type: DirectoryOrCreate
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/csi-xsky-nfs-driver
            type: DirectoryOrCreate

并创建 kubectl create -f deployment.yaml 成功后应该看到类似下面的容器

resizer_pod

接下来验证

由于我已经写好了自己的csi-nfs deriver此次省略csi-nfs driver的部署, 如果是其他的csi driver,请根据自己情况修改上面的resizer的yaml里面的配置已经下面的storageclass

6) 创建storageclass.yaml

sc的内容应该根据具体的driver,这里是我在用的csi-nfs的例子,注意添加一个字段allowVolumeExpansion: true

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-nfs-sc
provisioner: com.nfs.csi.xsky
parameters:
    xmsServers: 10.252.90.39,10
    user: admin
    password: admin
    shares: 10.252.90.123:/sdsfs/anyfile/
    clientGroup: ""
reclaimPolicy: Delete
allowVolumeExpansion: true

kubectl create -f storageclass.yaml

7)创建pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-nfs-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-nfs-sc

kubectl create -f pvc.yaml

成功后应该看到下面这种绑定的 bond

8) 创建pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: csi-nfs-demopod
  namespace: default
  labels:
    app: demo
spec:
  containers:
   - name: web-server
     image: nginx
     volumeMounts:
       - name: mypvc
         mountPath: /var/lib/www/html
  volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: csi-nfs-pvc
       readOnly: false

kubectl ctreate -f pod.taml

bond

9) 开始扩容

找到前面创建的pvc,这里的pvc名为csi-nfs-pvc 那就执行kubectl edit pvc csi-nfs-pvc 把大小改成10G doresize

10) 应用之后如无意外,resizer容器应有监听到pvc修改的事件 event

11) 如果driver层已经实现了扩容功能应该会收到扩容请求并进行处理 由于这里我们的driver还未完成实现已经测试该功能,因此这里看到pv大小没更新,但是实际存储端已经更新(果然是未经测试的代码.. updated

sds

如何实现

  1. 这里没有明确driver怎么实现resize,目前可以先返回空,只打个日志观察即可

后面代码的实现的时候可以慢慢测,具体使用的存储进行具体扩容处理(比如调相应的扩容接口)

如果是在controller服务中实现扩容,需要实现以下接口

//support only csi version is 1.11+
//totest
func (cs *controllerServer) ControllerExpandVolume(ctx context.Context, req *csi.ControllerExpandVolumeRequest) (*csi.ControllerExpandVolumeResponse, error) {
 log.Debugf("ControllerExpandVolume req[%#v]", req)
 newSize := 1000000
 //todo
 return &csi.ControllerExpandVolumeResponse{
  CapacityBytes: newSize,
  NodeExpansionRequired:false,
 }, nil
}
  1. 如果需要在node节点上执行扩容,那么需要实现在csi driver的NodeServer中实现
func (ns *nodeServer) NodeExpandVolume(ctx context.Context, req *csi.NodeExpandVolumeRequest) (*csi.NodeExpandVolumeResponse, error) {
 log.Debugf("NodeExpandVolume req[%#v]", req)
 volumePath := req.GetVolumePath()
//do resize on volumePath, e.g. format xxx
 log.Infof("NodeExpandVolume: resizefs successful volumeId: %s, devicePath: %s, volumePath: %s", volumeId, devicePath, volumePath)
 return &csi.NodeExpandVolumeResponse{}, nil
}
  1. 由于文件支持online resize,只需要实现ControllerExpandVolume接口即可