container toolkit daemon only cleans up containerd runtime config, and doesn't clean up the nvidia container runtime binaries on host /user/local/nvidia

[wyike]

I also see code here https://gitlab.com/nvidia/container-toolkit/container-toolkit/-/blob/main/tools/container/nvidia-toolkit/run.go?ref_type=heads#L131-140. It only reset the container runtime config.

After gpu operator or container toolkit removal, the nvidia/toolkit will be left on host:

worker/d3067738-6590-433d-8dd5-2568f7bccbb9:~# ls /usr/local/nvidia/toolkit/
libnvidia-container-go.so.1       nvidia-container-cli          nvidia-container-runtime.cdi.real   nvidia-container-runtime.legacy.real  nvidia-ctk.real
libnvidia-container-go.so.1.15.0  nvidia-container-cli.real     nvidia-container-runtime-hook       nvidia-container-runtime.real
libnvidia-container.so.1          nvidia-container-runtime      nvidia-container-runtime-hook.real  nvidia-container-toolkit
libnvidia-container.so.1.15.0     nvidia-container-runtime.cdi  nvidia-container-runtime.legacy     nvidia-ctk

Asking here if it is an intentional design or some cleanup step we have missed. Thanks!

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[elezar]

@wyike this is a known issue and is also "working as intended". If we were to remove the executables from the host, containers that have already started that were configured (e.g. through a runtime class) to use these would not be able to terminate cleanly.

@tariq1890 do you have thoughts on how to handle this particular issue in terms of triaging it?