背景#

在使用蓝盾「Docker公共构建机」一段时间后，我们发现构建镜像偶发性超时。排查后发现是由于集群的 Node 节点的磁盘满了，本文会介绍如何清理构建缓存。

我们发现构建镜像偶发性超时，排查发现是上了 Docker-in-Docker 构建镜像之后发生的，而且发生频率越来越高，进一步排查发现是由于 Pod 会通过 hostPath 挂载工作目录和日志目录，由于构建任务过多导致 Node 节点磁盘打满。

排查过程#

事件分析#

通过 Pod 事件可以发现是由于 Node 节点磁盘打满，导致 Pod 被驱逐，构建任务失败。

1
Events:
2
  Type     Reason                 Age                 From     Message
3
  ----     ------                 ----                ----     -------
4
  Warning  Evicted                20m                 kubelet  The node was low on resource: ephemeral-storage. Container build1753761077695-ivcpmoxg was using 1580320Ki, which exceeds its request of 0.
5
  Normal   NodeHasNoDiskPressure  3m (x32 over 6d5h)  kubelet  Node 10.10.32.2 status is now: NodeHasNoDiskPressure

pod yaml:

1
volumes:
2
- hostPath:
3
    path: /data/landun/workspace/build1753761077695-ivcpmoxg
4
    type: ""
5
  name: data-volume
6
- hostPath:
7
    path: /data/landun/logs/build1753761077695-ivcpmoxg
8
    type: ""
9
  name: logs-volume

是由于 Pod 通过 hostPath 挂载工作目录和日志目录，通过 hostPath 挂载目录是为了做缓存，当同一流水线任务重复执行时能够加速。

dispatch-k8s-manager 模块的配置文件: dispatch-k8s-manager/resources/config.yaml

1
dispatch:
2
  volume:
3
    # 构建机脚本
4
    builderConfigMap:
5
      name: dispatch-kubernetes-builder
6
      items:
7
        # 初始化脚本
8
        - key: initsh.properties
9
          path: init.sh
10
        # 登录调试需要的sleep脚本
11
        - key: sleepsh.properties
12
          path: sleep.sh
13
    hostPath:
14
      # 数据盘
15
      dataHostDir: /data/landun/workspace
16
      # 日志盘
17
      logsHostDir: /data/landun/logs
18
    # 应用数据使用cfs
19
    cfs:
20
      path: /data/cfs
21
  volumeMount:
22
    dataPath: /data/landun/workspace
23
    logPath: /data/logs
24
    builderConfigMapPath: /data/landun/config
25
    cfs:
26
      path: /data/bkdevops/apps
27
      readOnly: true

源码分析#

dispatch-k8s-manager/pkg/apiserver/service/builder_start.go

1
// getBuilderVolumeAndMount 获取一些构建机的常规的被挂载到pod上的volume和mount
2
func getBuilderVolumeAndMount(
3
  workloadName string,
4
  nFSs []types.NFS,
5
) (volumes []corev1.Volume, volumeMounts []corev1.VolumeMount) {
6
  volumes = getBuilderPodVolume(workloadName)
7
  volumeMounts = getBuilderPodVolumeMount()
8

9
  ...
10

11
  return volumes, volumeMounts
12
}
13

14
// getBuilderPodVolume 获取一些构建机的常规的被挂载到pod上的volume，包括配置configmap和data目录hostpath
15
func getBuilderPodVolume(workloadName string) []corev1.Volume {
16
  dataHostPath := filepath.Join(config.Config.Dispatch.Volume.HostPath.DataHostDir, workloadName)
17
  logHostPath := filepath.Join(config.Config.Dispatch.Volume.HostPath.LogsHostDir, workloadName)
18

19
  var items []corev1.KeyToPath
20
  for _, v := range config.Config.Dispatch.Volume.BuilderConfigMap.Items {
21
    items = append(items, corev1.KeyToPath{
22
      Key:  v.Key,
23
      Path: v.Path,
24
    })
25
  }
26

27
  return ...
28
}

通过源码分析可以发现 hostPath 是通过 dispatch-k8s-manager/resources/config.yaml 加上 workloadName 拼接而成的，所以没办法通过配置文件控制不使用 hostPath，于是我们通过定时任务来清理该缓存。

解决方案#

参考 bk-applog-bkapp-filebeat 的日志清理方案，通过 DaemonSet 实现蓝盾挂载工作目录实施定时清理操作。

1
NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
2
bk-applog-bkapp-filebeat-ingress       18        18        18      18           18          <none>          424d
3
bk-applog-bkapp-filebeat-json          18        18        18      18           18          <none>          424d
4
bk-applog-bkapp-filebeat-log-cleaner   18        18        18      18           18          <none>          424d
5
bk-applog-bkapp-filebeat-stdout        18        18        18      18           18          <none>          424d
6
bk-ci-builder-cleaner                  18        18        18      18           18          <none>          13d

编写 daemonSet.yaml

1
apiVersion: apps/v1
2
kind: DaemonSet
3
metadata:
4
  name: bk-ci-builder-cleaner
5
  namespace: blueking
6
  labels:
7
    app: bk-ci-builder
8
spec:
9
  revisionHistoryLimit: 10
10
  selector:
11
    matchLabels:
12
      app: bk-ci-builder
13
  template:
14
    metadata:
15
      labels:
16
        app: bk-ci-builder
17
      name: bk-ci-builder-cleaner
18
    spec:
19
      hostPID: true
20
      restartPolicy: Always
21
      serviceAccountName: bk-applog-bkapp-filebeat
22
      containers:
23
      - name: batch-delete-files
24
        image: xxx.xxx.com/bk-ci-builder-cleaner:v1
25
        imagePullPolicy: IfNotPresent
26
        command:
27
        - bash
28
        args:
29
        - -c
30
        - while true; do ./delete_files.sh; sleep 21600; done;
31
        resources:
32
          requests:
33
            cpu: 25m
34
            memory: 32Mi
35
          limits:
36
            cpu: 2560m
37
            memory: 256Mi
38
        volumeMounts:
39
        - mountPath: /data/devops/workspace
40
          name: data-volume
41
        - mountPath: /data/devops/logs
42
          name: logs-volume
43
      volumes:
44
      - name: data-volume
45
        hostPath:
46
          path: /data/landun/workspace
47
          type: DirectoryOrCreate
48
      - name: logs-volume
49
        hostPath:
50
          path: /data/landun/logs
51
          type: DirectoryOrCreate

缓存清理脚本 delete_files.sh

1
#!/usr/bin/env bash
2
# delete_files.sh  —— 正式删除版
3
# 同时扫描 /data/devops/workspace 和 /data/devops/logs
4
set -euo pipefail
5

6
# --------- 可配置参数 ---------
7
ROOT_DIRS=("/data/devops/workspace" "/data/devops/logs")
8
RETENTION_DAYS=7
9
LOG_FILE="/tmp/delete_build_dirs.log"
10
# -----------------------------
11

12
log() {
13
  printf '%s [%s] %s\n' "$(date '+%F %T')" "$1" "$2" | tee -a "$LOG_FILE"
14
}
15

16
cutoff_date=$(date -d "$RETENTION_DAYS days ago" +%F)
17

18
log INFO "==== 开始检查并删除 $RETENTION_DAYS 天未更新的 build* 目录 ===="
19

20
for root in "${ROOT_DIRS[@]}"; do
21
  [[ -d $root ]] || { log WARN "目录不存在: $root"; continue; }
22

23
  for dir in "$root"/build*; do
24
    [[ -d $dir ]] || continue
25

26
    # 二次确认：目录内是否仍无任何 7 天内更新的文件
27
    if ! find "$dir" -type f -newermt "$cutoff_date" -print -quit | grep -q .; then
28
        log DELETE "$dir"
29
        rm -rf "$dir"
30
    else
31
        log SKIP "$dir"
32
    fi
33
  done
34
done
35

36
log INFO "==== 清理完成，日志: $LOG_FILE ===="