A Forgejo-Runner with Podman on Alpine

What this is about

This is a guide on how to install a Forgejo-runner on alpine Linux using rootless Podman running as its own user.

Don't let the length of this post scare you. most of it is explaining why this guide works which should make catching mistakes easier.

I'll assume you have root access to the machine to install dependencies, service scripts and for creating users.

The installation here will use the runit service supervisor, as it is available on every distribution and easy to learn.

This guide is structured, with a future you, who has multiple runners (i.e. for another codeforge, organisation, friends, …) in mind. You'll find hints for repeating this guide.

We will:

• Install the needed dependencies
• Add a user and configure subuid and subgid
• Set up Podman as a service
• Configure the runner
• Set up the runner as a service
• Test the runner
• Enable Podman in Podman (optional)

Installing the Needed Dependencies

Community repository needed: Some of these packages come from the alppine community repository. Uncomment the relevant line in /etc/apk/repositories and run apk update to enable it.

apk add forgejo-runner podman slirp4netns runit sudo iptables

If you want to use Podman inside your CI: Add the fuse package if you want to enable Podman in Podman for your runner (i.e. for building containers).

The slirp4netns package is needed for networking with rootless Podman.

sudo is needed to switch users, we'll also use it as part of the runit services to make configuration easier.

iptables is needed by podman for networking.

Enabling Cgroups

Cgroups are needed by container runtimes like podman, to enable the on alpine:

# Start cgroups
/etc/init.d/cgroups start

# Make sure it gets started on every boot
rc-update add cgroups

# Should say that it is "started"
/etc/init.d/cgroups status

Note: Cgroups are a kernel feature, but alpine treats them as if they were a service to make them more convenient to manage.

Starting runit

After installation we have to tell OpenRC to start runit and do that on every boot:

# Start runit
/etc/init.d/runitd start

# Make sure it gets started on every boot
rc-update add runitd

# Should say that it is "started"
/etc/init.d/runitd status

Creating a User for the Runner

For the runner we'll create a regular user account without any special access. For this example I'm calling the user runner-for-someone, you can of course use your own name, make sure you always replace runner-for-someone with that.

adduser -D runner-for-someone

The -D option will disable password login for the just created user, there will be no need to log into it directly.

Use a naming convention: Using the runner-for-… naming scheme makes it easier to keep track of which user accounts are for runners and who the runners are for. It is not a requirement, but choosing a naming convention and sticking with it helps in the long run.

Now the important part is to add subuids and subgids.

Configure SubUIDs

In the file /etc/subuid add the following line to reserve 99999 user ids starting at user id 100000.

Important: If there are already UIDs reserved as subuids in that range, pick a multiple of 100000 where the next 99999 UIDs aren't reserved (i.e. 500000)).

Why multiples of 100000? Reserving multiples of 100000 isn't a requirement, but it makes it very easy to keep track of reservations.

runner-for-someone:100000:99999

Configure SubGIDs

And finally add the same line you added to /etc/subuid to /etc/subgid too.

Is reserving the same range a requirement? Using the same range for SubUIDs and SubGIDs isn't required, but again, it makes it easier to keep track of allocations.

Testing Podman

sudo -iu runner-for-someone podman run --rm alpine echo "It works!"

This should run the echo "It works!" command inside a rootless container that runs under the runner-for-someone user on the host.

Note on the sudo invocation: The -i flag means the command is ran as if it was for an interactive session, which sets up some extra environment variables that Podman needs. The -u runner-for-someone part tells sudo to switch to the given user instead of root.

If you get errors about insufficient SubUIDs and SubGIDs and have ran Podman before reserving the SubUIDs/SubGIDs the following command should make Podman aware of the change and fix it.

sudo -iu runner-for-someone podman system migrate

If you're here after creating a Podman service: Remember to restart Podman using sv restart /etc/service/runner-for-someone_podman for the fix to take effect in the runner.

Podman as a Service

Now that we know, that Podman is working we can start it as a service using runit.

Actually, to make your life easier should you want multiple runners (i.e. for yourself, an organisation, a second account, …), we'll create an easy to use template and then turn it into a service. One Podman service per user.

I though Podman doesn't require a service? The forgejo-runner wants to talk to Podman using a Unix socket, this is why wee need a Podman service.

The Podman Service Template

Note if you are creating your second runner: You only have to set up the templates once, if you already have set it up correctly you can skip to creating the service.

Create the directory /etc/sv/generic-forgejo-runner-podman/

and the file /etc/sv/generic-forgejo-runner-podman/run with the following content:

#!/bin/sh

# This parses the username from the directory name
USERNAME="$( basename "$(pwd)" | cut -d "_" -f 1 )"

# And this switches the user and runs the actual podman command
exec sudo -iu "$USERNAME" \
        sh -c 'exec podman system service --time=0 "unix://${HOME}/podman.sock"'

Then run the following commands to make the script executable:

chmod +x /etc/sv/generic-forgejo-runner-podman/run

What the Service Template Does

To explain what's going on (feel free to skip to the next headline):

This template will be usable without any configuration, any service created from it should have a name that is the username it runs as, the first component followed by an _podman.

USERNAME="$( basename "$(pwd)" | cut -d '_' -f 1 )"

This first line makes use of the fact that with runit we can rely on the working directory being the one that is named after out service. The basename "$(pwd)" outputs the name of the directory we are in without the path and the cut -d '_' -f 1 cuts the name at the underscore (_) and outputs what was before the underscore.

exec sudo -iu "$USERNAME" \

The sudo -iu "$USERNAME" switches into the user account whose name ended up in the USERNAME variable in the first line simulating an interactive login to get some extra environment variables that Podman expects to exist.

The exec at the start tells the shell to "replace" itself with the command instead of starting it as a child process. This is the simplest way to ensure that stopping the service will actually stop it and not leave some detached processes running in the background.

The backslash (\) at the end indicates that the next line is a continuation of this one instead of a new command. It is just there for readability.

        sh -c 'exec podman system service --time=0 "unix://${HOME}/podman.sock"'

This looks like a new command, but it is actually an argument to and ran by sudo in the context of the runner user.

The sh -c runs whatever comes next as if it was a little shell script, so the next argument is actually another command.

exec podman system service --time=0 "unix://${HOME}/podman.sock"

Again exec at the start so that the shell replaces itself instead of getting in the way.

And our actual podman system service command that makes Podman reachable using a Unix socket in the home directory.

The --time=0 tells Podman to not time out when there is nothing to do.

The variable ${HOME} gets evaluated now instead of in the previous steps because the whole command was wrapped in single quotes. This is important because only now $HOME is set to the runner users home instead of /root.

Why do this detour instead of putting the socket inside /home/$USERNAME/podman.sock even though that would have the same result?

Because /home/$USERNAME is only correct in the usual case, if you want to put the runner homes somewhere else in the future this will save quite some time and headaches.

Sometimes, being lazy requires a bit of effort.

And why $HOME/podman.sock instead of the default path?

Because that puts it in plain sight, less "magic" to get confused by.

Any other writable file path owned by the runner user would have worked too.

Creating the Actual Service

Template done, now lets create a service:

# Create the directory for the actual service. Note the USERNAME_podman pattern
mkdir /etc/sv/runner-for-someone_podman

# Take the run script from the template using a symbolic link
ln -s /etc/sv/generic-forgejo-runner-podman/run /etc/sv/runner-for-someone_podman/

# And create a symbolic link in /etc/service to "enable" the service
ln -s /etc/sv/runner-for-someone_podman /etc/service/

# The service should now be in an "up" state
sv status /etc/service/runner-for-someone_podman

You can control runit services using the sv command and through a path to the directory that contains the run script.

Have a look at the sv manual if you are not yet familiar with it

If you need it for debugging, you can always sv stop your service and run the run script like you would run any other shell script, make sure you are in the right directory so that you can run it as ./run so that the username gets read correctly.

Configuring the Forgejo Runner

For this step witch into the runner user using the following command:

sudo -iu runner-for-someone

Getting a Registration Token

Now is a good time to obtain a runner token …

… for yourself:

1. Log into the Forgejo instance using the website
2. Click on your avatar in the top right
3. Navigate to
   • Settings
   • Actions (in the Sidebar)
   • Runners
   • Create new runner (Button in the top right)
4. Copy the registration token

… for your organisation:

1. Log into the Forgejo instance using the website
2. Navigate to the repository overview of your organisation
3. Click the Settings link in the to right (Might be hidden inside a 3-dot menu)
4. Navigate to
   • Actions (in the Sidebar)
   • Runners
   • Create new runner (Button in the top right)
5. Copy the registration token

Leave that tab open, we will later use it to observe our runner.

Registering the Runner

Back in your shell run:

forgejo-runner register

It will interactively ask a few questions.

The resulting registration will be store in the file .runner, you don't have to do anything with it, but you should know where your configuration is.

If you refresh the browser window where you got the token from now, you should see a new runner that is marked a "Offline". (Leave the tab open, we'll need it again after creating the runner service)

Generating the Runner Configuration File

Speaking of configuration, your runner needs a configuration file.

Generate one:

forgejo-runner generate-config > ~/config.yml

You don't have to change this configuration file either, but you should have a look at it anyway.

The most interesting settings are:

enable_ipv6

If you know you have working IPv6 set this to true.

force_pull

Set this to true to make Podman check if a new version of an image is available every time it is needed. Podman is smart enough to not download an image again if it is already up to date.

container.docker_host

Leave it set to - or an empty string, the runner will then make use of the DOCKER_HOST variable to find the socket, which we can set in the service file.

privileged

Be careful with this one, if enabled container isolation becomes almost non-existent and everything that runs on you CI will have full access to your runner user. Leave it off.

How to quickly find out if IPv6 is working? Test if ping v6.echoip.slatecave.net or pinging any other IPv6 only host works.

We are done with the runner user for now.

exit

The Runner Service

Like with the Podman service this will be split into a template and an actual service.

Again, resulting in one service per user.

The Runner Service Template

Create a directory at /etc/sv/generic-forgejo-runner-runner/.

Create a file /etc/sv/generic-forgejo-runner-runner/run with the following content:

#!/bin/sh

USERNAME="$( basename "$(pwd)" | cut -d "_" -f 1 )"

# Make sure the podman service is running
sv start "../${USERNAME}_podman"

# Start the forgejo runner
exec sudo -iu "$USERNAME" \
	sh -c 'DOCKER_HOST="unix://${HOME}/podman.sock" exec forgejo-runner daemon'

Make sure the file is executable:

chmod +x /etc/sv/generic-forgejo-runner-runner/run

Explaining the Runner Service Template

This works much like the template we used to start Podman, so I'll only go over the parts that are different.

sv start "../${USERNAME}_podman"

This will make sure the corresponding Podman service has started before starting the runner, even if it was stopped for some reason. sv start will by default wait for up to 7 seconds which should be plenty of time for Podman to start.

And inside the sh -c:

DOCKER_HOST="unix://${HOME}/podman.sock" exec forgejo-runner daemon

This will set the DOCKER_HOST environment variable and run the forgejo-runner in its service mode. The exec again are there to get the shells out of the way as soon as they've done their work.

Not really a daemon: Historically the "daemon" mode made a program detach itself from the shells process tree and run in the background, which works great for unmanaged services, but actually gets in the way of supervisors like runit. The forgejo-runner daemon command doesn't do that, contrary to what the name might suggest.

Creating the Actual Runner Service

Almost the same commands as with Podman to turn the template into a running service:

# Create the directory for the actual service. Note the USERNAME_runner pattern
mkdir /etc/sv/runner-for-someone_runner

# Take the run script from the template using a symbolic link
ln -s /etc/sv/generic-forgejo-runner-runner/run /etc/sv/runner-for-someone_runner/

# And create a symbolic link in /etc/service to "enable" the service
ln -s /etc/sv/runner-for-someone_runner /etc/service/

# The service should now be in an "up" state
sv status /etc/service/runner-for-someone_runner

Testing the Runner

Refresh the tab with the runner overview, the status should have changed from "Offline" to "Idle".

Now create a new Forgejo repository, make sure the actions tab is enabled, if not click on Settings in the upper right, navigate to Units, Overview, enable the checkbox labelled Actions and click Save.

Create a file in .forgejo/workflows/test.yaml in the repository:

---
on:
  push:
jobs:
  test:
    runs-on: docker
    container:
      image: alpine
    steps:
      - run: echo "Hello World!"

Commit and push it.

In the "Actions" tab there should now be an action that is either waiting or already running. Don't worry if it takes up to a minute for the runner to pick up the job.

After a few seconds everything should go green with a "Hello World!" in the logs. If it doesn't there is usually a hint hidden in the logs as to what went wrong.

If it fails with "failed to create container": This one is misleading, try to sudo -iu runner-for-someone podman pull alpine, it'll output a more helpful error message. (Also have a look at testing Podman again.)

🎉 Congratulations, you now have a working runner.

You should keep the test repository, it'll stay useful.

If you now want to try out some premade using actions remove the image: alpine line. Using alpine Linux as a base image is great if you know exactly what you need and want an efficient CI. But most premade actions unfortunately rely on the Ubuntu based nodejs image, which is the reason for that one being the default.

Enabling Podman in Podman

To enable Podman in Podman functionality without having to fall back on dropping all isolation the containers need access to fuse (install and modprobe it if you haven't done already) and some additional capabilities to be able to create a container environment inside the container.

The Forgejo runner doesn't directly support this way of running containers yet, but since we have a user just for the runner configuring Podman directly works too:

# Install and immedeately enable fuse
apk add fuse
modprobe fuse

# switch to the runner user
sudo -iu runner-for-someone

# create the directory for the configuraion file
mkdir -p ~/.config/containers/

Then create the file ~/.config/containers/containers.conf:

[containers]
devices = ["/dev/fuse"]
default_capabilities = [
        "SYS_ADMIN",
        "MKNOD",
        {append=true}
]

And to test if that configuration file worked:

podman run --rm	quay.io/podman/stable podman run alpine echo Hello Podman

If it didn't work you'd either get a message that mentions the absence of fuse or "potentially insufficient UIDs or GIDs" if the SYS_ADMIN capability is missing.

To also enable this feature in the runner you have to restart the Podman service:

# Back to root
exit

# Restart the podman serivce
sv restart /etc/service/runner-for-someone_podman

If you want to know what exactly why those are necessary I highly reading the ultimate "Podman inside a container" guide by Dan Walsh and Urvashi Mohnani on the Redhat blog:

How to use Podman inside of a container

Maintenance

It is a good idea to get rid of old images from time to time, you can use the following commands to manage Podman images for your runner:

sudo -iu runner-for-someone

# list images
podman images

# remove outdated or no longer needed images
podman image rm <image-id-goes-here>

Another thing to check for are stopped containers that never got cleaned up, this may happen because of sudden shutdowns or by forgetting the --rm on containers during testing.

sudo -iu runner-for-someone

# list running AND stopped containers
podman ps --all

# remove old containers
podman rm <container-id-goes-here>

Other Resources

• Forgejo Actions user guide
• Forgejo Actions administrator guide
• Forgejo Runner installation guide
• Using Forgejo Actions (Self-hosted) by Codeberg

The Beginning …

Now you have your own runner (hopefully 😅), it's up to you what you want to with it.

I highly encourage you to keep the test repository around, try some unconventional things, break it, fix it, DIY some actions instead of importing and learn what makes the actions run.

Another thing to try is building your own images.

Whatever you do, have fun and make the world a nicer place to be!

License Note

This article is licensed under CreativeCommmons BY-SA 4.0 International (this is different from the usual CC BY-NC-SA here).

If possible please link back to this article.