It all starts on the Start method (Ha!). Something somewhere sends a value into the channel to conciliate the actual state of the pod with the desired state. I left out the code that sends this value because it's irrelevant to our investigation.

// in Start on status_manager.go
go wait.Forever(func() {
  for {
    select {
      case syncRequest := <-m.podStatusChannel:
        klog.V(5).InfoS("Status Manager: syncing pod with status from podStatusChannel",
          "podUID", syncRequest.podUID,
          "statusVersion", syncRequest.status.version,
          "status", syncRequest.status.status)
        m.syncPod(syncRequest.podUID, syncRequest.status)
     }
  }
}

// in syncPod on kubelet.go
// pcm is a podContainerManagerImpl struct
if !pcm.Exists(pod) && !firstSync {
  p := kubecontainer.ConvertPodStatusToRunningPod(kl.getRuntime().Type(), podStatus)
  if err := kl.killPod(pod, p, nil); err == nil {
    podKilled = true
  } else {
    klog.ErrorS(err, "KillPod failed", "pod", klog.KObj(pod), "podStatus", podStatus)
  }
}

Here, it checks if the pod still exists. If not, it calls killPod, telling containerd to kill it. Notice how there is no log here stating the exact reason why the pod was killed. This makes things challenging to troubleshoot.

Here is the part where the check happens:

// in cgroup_manager_linux.go
// pcm will call this method
func (m *cgroupManagerImpl) Exists(name CgroupName) bool {
        return m.Validate(name) == nil
}

func (m *cgroupManagerImpl) Validate(name CgroupName) error {

  if libcontainercgroups.IsCgroup2UnifiedMode() {
    cgroupPath := m.buildCgroupUnifiedPath(name)
    neededControllers := getSupportedUnifiedControllers()
    enabledControllers, err := readUnifiedControllers(cgroupPath)
    if err != nil {
      return fmt.Errorf("could not read controllers for cgroup %q: %w", name, err)
    }
    difference := neededControllers.Difference(enabledControllers)
    if difference.Len() > 0 {
      return fmt.Errorf("cgroup %q has some missing controllers: %v", name, strings.Join(difference.List(), ", "))
    }
    return nil // valid V2 cgroup
  }
  // Rest of cgroups v1 logic
}

kubelet parsed the cgroup of this pod as kubepods-burstable-<pod_id>.slice inside kubepods-burstable.slice. I grepped the PID of the container in /sys/fs/cgroup and found that it was in cgroup kubepods-besteffort-<pod_id>.slice:cri-containerd:<container_id> inside system.slice. The container cgroup was not related at all to the pod cgroup.

With the broken version, the best effort cgroup has the following configuration (same with burstable):

/sys/fs/cgroup/
├── kubepods.slice
│   ├── kubepods-besteffort.slice
│   │   └── kubepods-besteffort-pod<pod_id>.slice
├── system.slice
│   ├── kubepods-besteffort-pod<pod_id>.slice:cri-containerd:<container_id>

With the systemd option working correctly:

/sys/fs/cgroup/kubepods.slice/
├── kubepods-besteffort.slice
│   └── kubepods-besteffort-pod<pod_id>.slice
│       └── cri-containerd-<container_id>.scope

One thing that I thought it was strange is that the pod cgroup does exist in the v2 format. Putting a log with the error there surprised me with the following message:

cgroup [\"kubepods\" \"besteffort\" \"pod7149273f-1369-42ff-ae1f-79b1529bba7b\"] has some missing controllers: cpuset

Okay. kubelet identifies the cgroup as missing because it's missing a controller. Who's removing the cpuset controller then?

Let's digress a bit. Kubernetes assigns a pod in one of the three Qualify of Service (QoS) classes. It sets different CPU scheduling and decides who will die first in case of memory pressure. The three types are:

• Guaranteed: Pods that are strict about their CPU and memory limit and requests
• Burstable: Pods that are less strict but still define at least one limit or request in one of its containers
• BestEffort: Pods that don't specify any limit or requests

kubelet uses the cpu.weight file to allocate CPU time for the given processes based on their QoS. This calculation happens every minute, and in the end, it will send a D-Bus message to systemd with the CPUWeight property to the QoS cgroup. kubelet sets the minimum share of one to the BestEffort cgroup and calculates the BestEffort shares based on the existing requests of active pods. As a good citizen, kubernetes rewards the pod with more CPU time. So, always specify a limit and request in your pod definition.

What does this have to do with the previous error? I noticed that the pod died a couple of seconds after kubelet sent this CPUWeight request every minute. Looking at the files of the QoS cgroup, I could see that the cgroup.subtree_control was temporarily missing the cpuset controller. cgroups uses this file to block or allow what controllers the children can access.

> cd /sys/fs/cgroup
> mkdir -p custom_parent/custom_child
> ls -l custom_parent/custom_child/cpu*
2
> cat custom_parent/custom_child/cgroup.controllers
# Nothing is returned
> echo +cpu > custom_parent/cgroup.subtree_control
> cat custom_parent/custom_child/cgroup.controllers
cpu
> ls -l custom_parent/custom_child/cpu* | wc -l
5

The cpu files, like cpu.weight or cpu.max are only accessible when adding the cpu controller to cgroup.subtree_control.

This is a demo of what's happening every minute. I called systemd directly, so it's more deterministic, but it's the same operations under the hood.

Notice how after a couple of seconds, kubelet will add the cpuset controller to get back to "normality". This happens because runc will try to create the cgroup via file API after sending the D-Bus messages to systemd. Not sure if it's just to guarantee that the cgroup is created correctly. By the way, it's the kubelet who uses runc as a library to create the QoS cgroup, not containerd.

So, one crucial detail is that something removes the controller only when using the systemd API. It's still there when writing to cpu.weight directly. So it's probably not the kernel messing up with the controller.

Issuing a strace -p 1, I found out that systemd was the process removing the cpuset in the cgroup.subtree_control file of the QoS cgroup.

systemd doesn't remove the controller in the working setup because runc adds the cgroup via systemd with the Delegate option: This option, according to the docs, does something relevant for us:

systemd won't fiddle with your sub-tree of the cgroup tree anymore. It won't change attributes of any cgroups below it, nor will it create or remove any cgroups thereunder, nor migrate processes across the boundaries of that sub-tree as it deems useful anymore.

So, with Delegate as a children of the pod and Qos cgroups, runc says: "Fuck off, systemd. I know what I'm doing. This cgroup and all its parents belong to me". systemd will reply: "Okay. Carry on. I will leave you alone". When there is no delegated cgroup, systemd will say: "Aha. All of these cgroups belong to me now. I will do whatever I want with it!".

The tricky question nagging me is: "Why is systemd removing this controller in the first place?". Following what systemd is doing is unsurprisingly hard to follow. Linux introduced cpuset in v2, and the same feature was in the cpu controller for v1. Perhaps it's not considering it when it restarts the controllers of non-delegated cgroup. I'm probably not seeing the forest for the trees and I leave it as an exercise for the reader to find out =P.

In the end, this "bug" might be a feature; otherwise, kubelet wouldn't restart the pod, and I would think the control plane was healthy.

A short summary of the broken version events:

1. containerd will create the container cgroup outside of kubepods-<qos>-<pod_id>.slice
2. kubelet sends a D-Bus message to systemd to change the CPUWeight property of the QoS cgroup
3. systemd writes this value to cpu.weight file
4. systemd removes the cpuset controller for whatever reason
5. kubelet will try to sync the pod and realize that a controller is missing
6. kubelet kills the pod because it thinks the cgroup is "gone."
7. kubelet "syncs" the QoS cgroup again and adds the cpuset via file API
8. the pod is up
9. go back to 1. the control plane is broken

The version with SystemdCgroup in containerd config:

1. containerd creates the container cgroup inside the pod cgroup with the Delegate option
2. kubelet sends a D-Bus message and systemd writes the value to cpu.weight file
3. systemd won't mess up with the parents of the delegated container cgroup
4. kubelet doesn't kill the pod because the cpuset controller is still there
5. the control plane is healthy

In my opinion, it's kubelet's responsibility to not allow the container manager and runtime to use a different cgroup driver. For instance, kubelet returned an error when docker didn't match the driver. Since 1.22, kubelet removed the docker integration, and it only supports managers implementing the CRI now. Apparently, the CRI specification doesn't provide an agnostic way to identify the cgroup driver of the container runtime. I'm still unable to see the whole picture yet, and the best way to avoid people shooting themselves in the foot.