Oh that's nice, would it be possible to run containerd only, without running any Kubernetes controllers? I suspect the controllers are the root cause of the huge resource consumption, because they are constantly checking for the system components and trying to reconcile them.

Yep! If you’re on MacOS check out Lima, it’s a project by Akihiro Suda who is one of the main contributors to nerdctl (and the original author iirc)

Rancher Desktop macOS version is based on Lima.