enable ai-services to run as non-root-user #267

Shubhamag12 · 2026-01-16T05:34:16Z

Context: https://ibm.ent.box.com/file/2102027246213

Description

Problem Statement

Currently, ai-services only supports execution as the root user.

Solution

This PR enables ai-services to run as a non-root user. The tool now supports a hybrid approach:

bootstrap (validate/configure): Runs with sudo for one-time system setup
runtime operations (application commands): Run as a regular user without elevated privileges

User Setup Requirements

To use ai-services as a non-root user, the following one-time setup is required:

1. Add user to sudo group (wheel)

sudo usermod -aG wheel <username>

This grants the user permission to execute specific commands with sudo when needed.

2. Enable systemd user session persistence

sudo loginctl enable-linger <username>

This ensures the user's systemd instance persists even when not logged in, which is required for:

Rootless podman socket to remain active
User services to run in the background

Implementation Details

Rootless Podman

Containers run as the regular user on the host (not root)
Uses user namespaces: processes appear as "root" inside containers but are actually the user's UID on the host
Podman socket created at /run/user/<UID>/podman/podman.sock (user-specific, not system-wide)

Directory Permissions

Bootstrap creates directories with ownership set to $SUDO_USER
Regular user has full read/write access to application directories
No root privileges required for runtime operations

Environment Variables

XDG_RUNTIME_DIR: Automatically set if missing (required for user systemd operations)
SUDO_USER: Used during bootstrap to determine the actual user for ownership settings

Q&A

Q: Why do we need to add the non-root user to the sudo group?
A: We still need certain commands to be executed with sudo, like ai-services bootstrap, as it configures and modifies the env which requires root permissions.

Q: Which commands would be run without sudo?
A: Except bootstrap, every other command would be run without sudo.

mayuka-c · 2026-01-20T03:16:35Z

ai-services/internal/pkg/validators/root/root.go

-		return fmt.Errorf("current user is not root (EUID: %d)", euid)
+	if euid != 0 && os.Getenv("XDG_RUNTIME_DIR") == "" {
+		uid := os.Getuid()
+		logger.Infoln("running command as %s", uid, logger.VerbosityLevelDebug)


Do we need to set the userID in XDG_RUNTIME_DIR?. Also a query on why do we need to set this?

From what I checked online, they say not to explicitly set this value tho (I dont know much on this :))

Usually it is automatically set, but I noticed in one LPAR where it was not set and that is why we configure it if not already done

XDG_RUNTIME_DIR points to the user specific runtime files.

systemctl --user needs to connect to the user's systemd instance and without this variable, systemctl doesn't know where to find the socket and hence we get "No medium found" error.

does setting userID in XDG_RUNTIMR_DIR has any implications?

okay, looks like its not needed if we are enabling linger, that is, loginctl enable-linger user, which is required and will be mentioned in the documentation

will remove this from the code and push the changes

wait, my bad, while creating the application we get failed to pull image registry.redhat.io/rhaiis/vllm-spyre-rhel9:3.2.5: reading JSON file "/run/containers/1001/auth.json": open /run/containers/1001/auth.json: permission denied because it is not able to lookup the directory.
So yes, this is required.

Apologies for creating the confusion.

mayuka-c · 2026-01-20T03:19:22Z

ai-services/internal/pkg/runtime/podman/podman.go

+func userPodmanURI(uid int) string {
+	return fmt.Sprintf(
+		"unix:///run/user/%d/podman/podman.sock",
+		uid,
+	)
+}


Just a check. Did all the pods deployment worked fine with non root?
Did you test by deploying the RAG application?

I tried this on a 3 spyre card setup, where all pods were in healthy state except vllm as it required 4 cards for instruct and 1 for reranker. I configured instruct to use 2 cards and hence it was unhealthy

ai-services/cmd/ai-services/cmd/bootstrap/configure.go

mayuka-c · 2026-01-20T03:28:18Z

ai-services/cmd/ai-services/cmd/application/create.go

+	cmd := exec.Command("sudo", "ppc64_cpu", "--smt")
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("failed to check current SMT level: %v, output: %s", err, string(out))
+	}


If we are using sudo, will this prompt for the password to be entered by user here?

if passwordless user is enabled, then no
else for all sudo operations, it will ask for password

but I think here it might fail, as exec.Command doesn't attach to terminal stdin by default afaik. Let me confirm and post results here

but I think here it might fail, as exec.Command doesn't attach to terminal stdin by default afaik.

confirmed this, I have disabled the spinner for smt level, since we might need user input for this

We need to enable sudo user without password so that there wont be any user intervention needed

I dont think we should do that
Its upto admin on how the user is setup, passwordless or with password

So now create command also requires SUDO_USER right because of setting smt level?.
Wouldnt this defeat the purpose of what we wanted to achieve if I'm correct.

We wanted bootstrap to be run by either root or sudo user and the create by a normal user right?

User will anyway have permissions to run as sudo, and since moving it to configure.go cannot be possible as we want to keep this smt value configurable as per application, thats why thought of adding sudo.
CC: @mkumatag

Let me know if there is any other alternative

Not sure. Maybe we can have a default constant value set I guess during bootstrap.
For now, I guess we need to document that only root user and non root users with sudo privileges can run.

yes, SMT will set per application template need

ai-services/internal/pkg/cli/helpers/models.go

ai-services/cmd/ai-services/cmd/bootstrap/configure.go

Signed-off-by: sagarwal-ibm <sagarwal@ibm.com>

mkumatag

initial comments, overall flow looks really complicated, might need some more time to go through this code and the description.

ai-services/cmd/ai-services/cmd/application/create.go

mkumatag · 2026-01-27T05:13:47Z

ai-services/cmd/ai-services/cmd/application/create.go

+	cmd := exec.Command("sudo", "ppc64_cpu", "--smt")
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("failed to check current SMT level: %v, output: %s", err, string(out))
+	}


yes, SMT will set per application template need

ai-services/cmd/ai-services/cmd/application/create.go

ai-services/cmd/ai-services/cmd/bootstrap/configure.go

ai-services/internal/pkg/runtime/podman/podman.go

Signed-off-by: sagarwal-ibm <sagarwal@ibm.com> address comments

mkumatag · 2026-02-02T14:46:40Z

please squash and merge the changes post @mayuka-c's approval.

mayuka-c · 2026-02-03T13:34:31Z

@Shubhamag12 is currently testing it. Once done we can merge

Shubhamag12 changed the title ~~Non root user~~ enable ai-services to run as non-root-user Jan 16, 2026

Shubhamag12 force-pushed the non-root-user branch 2 times, most recently from 3a07de7 to eea51b8 Compare January 16, 2026 08:27

Shubhamag12 requested review from mayuka-c and mkumatag January 16, 2026 09:02

Shubhamag12 marked this pull request as ready for review January 16, 2026 09:21

Shubhamag12 force-pushed the non-root-user branch from ff36007 to 13f1f24 Compare January 16, 2026 09:26

Shubhamag12 requested a review from manju956 January 19, 2026 09:06

mayuka-c reviewed Jan 20, 2026

View reviewed changes

ai-services/internal/pkg/cli/helpers/models.go Show resolved Hide resolved

Shubhamag12 force-pushed the non-root-user branch from 5d7ed99 to 9be37ea Compare January 20, 2026 08:28

manju956 reviewed Jan 21, 2026

View reviewed changes

ai-services/cmd/ai-services/cmd/bootstrap/configure.go Show resolved Hide resolved

enable ai-services to run as non-root user

c689173

Signed-off-by: sagarwal-ibm <sagarwal@ibm.com>

Shubhamag12 force-pushed the non-root-user branch from 9b6d840 to 61702bb Compare January 27, 2026 05:40

mkumatag requested changes Jan 27, 2026

View reviewed changes

fix lints

00741bf

Signed-off-by: sagarwal-ibm <sagarwal@ibm.com> address comments

Shubhamag12 force-pushed the non-root-user branch from 61702bb to 00741bf Compare January 27, 2026 09:47

mkumatag approved these changes Feb 2, 2026

View reviewed changes

Merge branch 'main' into non-root-user

b1a6a8e

Shubhamag12 added do not merge testing labels Feb 3, 2026

enable ai-services to run as non-root-user #267

Are you sure you want to change the base?

enable ai-services to run as non-root-user #267

Conversation

Shubhamag12 commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Problem Statement

Solution

User Setup Requirements

1. Add user to sudo group (wheel)

2. Enable systemd user session persistence

Implementation Details

Rootless Podman

Directory Permissions

Environment Variables

Q&A

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mayuka-c Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mkumatag left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mkumatag commented Feb 2, 2026

Uh oh!

mayuka-c commented Feb 3, 2026

Uh oh!

Reviewers

Assignees

Shubhamag12 commented Jan 16, 2026 •

edited

Loading

mayuka-c Jan 21, 2026 •

edited

Loading