website/content/blog/2022-06-02-zombies.md

+++ title = "[diagnostic] Zombies created by Gitea" date = 2022-06-02 description = "An increasing number of zombies processes are created by Gitea because it only kills its direct children on timeout." [taxonomies] tags = ['hostea', 'gitea', 'troubleshoot', 'problem']

[extra] author = 'dachary' +++

The first issue about zombie processes created by Gitea was reported in 2017 and resurfaced on a regular basis. Although it does not look pretty, zombie processes are leftovers that do not consume resources and never caused any kind of harm. Here is one scenario that will create a zombie:

• Gitea updates a mirror by spawning the process git remote update
• git remote update spawns yet another process, git fetch
• git fetch is stuck, for instance because of network problems, and Gitea eventually times out
• Gitea kill the process git remote update
• When killed git remote update does not kill its own child and git fetch becomes an orphaned process which keeps running
• When git fetch eventually completes it becomes a zombie because its original parent is no longer around to wait on it

PID 1 process and waiting on orphans

This scenario is not unique to Gitea and it is such a common pattern that safeguards have been implemented to mitigate the proliferation of zombies. Orphaned process are automatically attached to the process with PID 1, which is expected to wait on every process, whether it created them or not. When Gitea is installed from binary on GNU/Linux this is /bin/init and when Gitea runs from the default docker image this is s6: they will both wait on orphaned processes and there won't be any zombies.

What if Gitea is the only running process?

But when Gitea runs from the rootless docker image, Gitea is the only process running in the container. Orphaned processes will have Gitea as a parent but will not wait on them and they will stay in a zombie state forever. To reproduce this problem in a minimal way:

$ docker run --name gitea -p 8080:3000 -e GITEA__security__INSTALL_LOCK=true -d gitea/gitea:1.16.8-rootless
$ docker exec --user 1000 gitea gitea admin user create --admin --username root --password admin1234 --email root@example.com

The git command can then be replaced with a script that waits forever:

$ ( echo -e '#!/bin/bash\nsleep infinity' ) | docker exec -i --user root gitea tee /usr/bin/git
$ docker exec --user root gitea chmod +x /usr/bin/git

Trying to create a repository from the web interface will create the conditions for a zombie to show:

$ docker exec gitea ps -o ppid,pid,comm,args 
PPID  PID   COMMAND          COMMAND
    0     1 gitea            /usr/local/bin/gitea -c /etc/gitea/app.ini web
    1    94 sleep            [sleep]
    1    99 sleep            [sleep]
    1   111 sleep            [sleep]
    1   164 git              {git} /bin/bash /usr/bin/git -c credential.helper= -c protocol.version=2 -c uploadpack.allowfilter=true -c uploadpack.allowAnySHA1InWant=true init --bare
  164   165 sleep            sleep infinity

When the git process is killed by Gitea, the sleep child will be orphaned:

$ docker exec gitea ps -o ppid,pid,comm,args 
PPID  PID   COMMAND          COMMAND
    0     1 gitea            /usr/local/bin/gitea -c /etc/gitea/app.ini web
    1    94 sleep            [sleep]
    1    99 sleep            [sleep]
    1   111 sleep            [sleep]
    1   165 sleep            sleep infinity

Killing it will turn it into a zombie:

$ docker exec gitea kill 165
$ docker exec gitea ps -o ppid,pid,comm,args 
PPID  PID   COMMAND          COMMAND
    0     1 gitea            /usr/local/bin/gitea -c /etc/gitea/app.ini web
    1    94 sleep            [sleep]
    1    99 sleep            [sleep]
    1   111 sleep            [sleep]
    1   165 sleep            [sleep]

Killing a child process and all its children

There should be no need for an admin running Gitea to worry about those gory details, it should be taken care of regardless of the environment Gitea runs in. Fortunately there is a very simple way to avoid the creation of zombies by ensuring that all Gitea child process are process group leaders. In a nutshell it means that when the child is killed all its children and grand children are also killed.