Reap the container cgroup when the runc shim is stopped - #14220

Reap the container cgroup when the runc shim is stopped

Background

We run containerd on bare metal Kubernetes nodes and found nodes where containerd took 30+ seconds to start, every time it was restarted. Each start logged the same cleanup failures for the same set of old containers:

level=info  msg="cleaning up dead shim" id=<id> namespace=k8s.io
level=warning msg="failed to clean up after shim disconnected" error="unmount rootfs /run/containerd/io.containerd.runtime.v2.task/k8s.io/<id>/rootfs: device or resource busy"
level=error msg="failed to delete dead shim" error="signal: killed" id=<id>
failed to remove ... cri-containerd-<id>.scope: device or resource busy

The bundles of these containers could never be deleted. Every containerd start loaded them again, ran the same failing cleanup (with its unmount retries, until the 5s dead shim cleanup timeout), and gave up. On containerd 1.6/1.7, which loads shims serially, this adds about 5s of boot time per leaked bundle. On 2.x it happens in parallel, but the bundles, cgroups and processes still leak forever.

On such a node, the container cgroup still held processes (in some cases a frozen runc:[2:INIT]). They pinned the rootfs mount, so the unmount failed with EBUSY. Bundle.Delete() then failed, and the dead shim stayed around for the next start.

Root cause

manager.Stop() in the runc shim is what containerd runs to clean up after a shim that has died, and after a task whose creation failed. For tearing the container down, it relies entirely on runc delete --force. That is not enough:

• runc delete --force can fail. The failure is only logged, and cleanup continues with the rootfs unmount, which then fails because the processes are still there.
• runc delete --force returns success without doing anything when the container's state.json was never written. That is exactly the state a runc create killed partway through leaves behind, for example when containerd restarts (which kills its in-flight runc children) during NewTask. At that point runc has already created the cgroup and started runc init, and it may have left the cgroup frozen. This is Preventing containers from being unable to be deleted opencontainers/runc#4757 (merged, then reverted in Zombies with exec probes : Kubernetes 1.20-containerd 1.5.0-beta3 #5153 because of a conmon regression; Prepare v1.5.0-rc.0 #5257 is a new attempt that is still under discussion).

In both cases the processes left in the container cgroup keep the cgroup populated and the rootfs busy, and nothing ever reaps them.

Fix

After runc delete --force, whatever it reported, Stop() now reaps the container cgroup directly: it kills whatever is left in it, waits for it to empty, and removes it.

• The cgroup path comes from linux.cgroupsPath in the bundle's config.json. containerd writes that file before runc is ever invoked, so this does not depend on any runc state.
• The runtime's SystemdCgroup option decides whether the path uses systemd slice:prefix:name notation or is a literal cgroupfs path, the same way runc decides.
• cgroup v2: it writes cgroup.kill (which also kills frozen processes). On kernels older than 5.14 it falls back to the same bounded loop as v1.
• cgroup v1 (and the v2 fallback): each round freezes the cgroup, lists its processes, SIGKILLs them, and thaws the cgroup before waiting, because frozen tasks do not act on SIGKILL until thawed. The thaw also unsticks a cgroup that a killed runc create left frozen. Signals go through pidfds, re-checked against the cgroup membership, so a reused PID outside the cgroup is never signalled.
• The whole reap is bounded (2s), since it runs during containerd startup.
• When runc did its job, the cgroup is already gone and this is a no-op.

This kills nothing that runc delete --force is not already meant to kill: runc treats the container cgroup as owned by the container.

The fix lives in the shim because Stop() is the one place that runs in both cleanup paths: the create failure and the dead shim cleanup at boot. It also does not depend on how runc eventually addresses #4757. It covers both the state.json gap and a plain runc delete failure.

How to reproduce

I wrote a small, deterministic reproducer: https://github.com/yongxiu/leakrepro

It creates a container through the containerd client with an OCI createRuntime hook. runc runs that hook after it has created the cgroup and mounted the rootfs, but before it writes state.json. The hook freezes the container cgroup and SIGKILLs runc create. That leaves exactly the production end state:

• a frozen runc:[2:INIT], reparented to PID 1, still in the container cgroup and pinning the rootfs;
• no state.json, so runc delete --force exits 0 and does nothing.

No image or snapshotter is needed. It reproduces 100% of the time.

On a throwaway node (it deliberately leaks a process):

leakrepro create --address /run/containerd/containerd.sock --id leakrepro1
leakrepro inspect --id leakrepro1
systemctl restart containerd
journalctl -u containerd | grep "successfully booted"

The repo also has run-repro.sh. It runs each containerd build in a privileged, throwaway Docker container with its own containerd, so the host is not touched.

Results

runc 1.3.6, cgroup v2, 3 runs each, identical every run:

I also verified it on a real cgroup v2 node running containerd v2.2.1, with only this change cherry-picked into the shim. With the stock shim the container leaks. With the patched shim, leakrepro inspect reports CLEAN right after the failed create, and containerd restarts stay fast. After the fix there is no leak at all.

Tests

cgroup_linux_test.go adds unit tests for the path resolution (systemd notation including the root slice -.slice, cgroupfs paths with colons) and the choice of cgroup v1 subsystem. It also runs real-kernel tests against a delegated child cgroup: a live process, a frozen process, empty and already-removed cgroups, the bounded freeze-and-kill fallback, and a check that a listed PID that is no longer in the cgroup is not signalled. Those tests skip themselves when cgroup v2 delegation is not available.

Related: opencontainers/runc#4757, opencontainers/runc#5153, opencontainers/runc#5257