From 52b11b2bacc653d0abd2693b484cf32c69320f1e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lo=C3=AFc=20Dachary?= Date: Thu, 2 Jun 2022 10:52:01 +0000 Subject: [PATCH] new deploy: 2022-06-02T10:52:01+00:00 --- blog/atom.xml | 72 ++++++- blog/index.html | 35 ++++ blog/zombies/index.html | 360 +++++++++++++++++++++++++++++++++++ search_index.en.js | 2 +- sitemap.xml | 4 + tags/gitea/atom.xml | 72 ++++++- tags/gitea/index.html | 26 +++ tags/hostea/atom.xml | 72 ++++++- tags/hostea/index.html | 26 +++ tags/index.html | 8 +- tags/problem/atom.xml | 72 ++++++- tags/problem/index.html | 26 +++ tags/troubleshoot/atom.xml | 72 ++++++- tags/troubleshoot/index.html | 26 +++ 14 files changed, 863 insertions(+), 10 deletions(-) create mode 100644 blog/zombies/index.html diff --git a/blog/atom.xml b/blog/atom.xml index a5a9c02..bcbaa87 100644 --- a/blog/atom.xml +++ b/blog/atom.xml @@ -4,8 +4,78 @@ Zola - 2022-05-28T00:00:00+00:00 + 2022-06-02T00:00:00+00:00 https://hostea.org/blog/atom.xml + + [solved] Zombies created by Gitea version <= 1.16.8 + 2022-06-02T00:00:00+00:00 + 2022-06-02T00:00:00+00:00 + + https://hostea.org/blog/zombies/ + <p><strong>TL;DR: run Gitea version &gt;= 1.16.9 to avoid zombies</strong></p> +<p>The first <a href="https://github.com/go-gitea/gitea/issues/3242">issue about zombie processes</a> created by Gitea was reported in 2017 and <a href="https://github.com/go-gitea/gitea/issues/13987">resurfaced</a> on a <a href="https://github.com/go-gitea/gitea/issues/19077">regular basis</a>. Although it does not look pretty, zombie processes are leftovers that do not consume resources and never caused any kind of harm. Here is one scenario that will create a zombie:</p> +<ul> +<li>Gitea updates a mirror by spawning the process <code>git remote update</code></li> +<li><code>git remote update</code> spawns yet another process, <code>git fetch</code></li> +<li><code>git fetch</code> is stuck, for instance because of network problems, and Gitea eventually times out</li> +<li>Gitea kill the process <code>git remote update</code></li> +<li>When killed <code>git remote update</code> does not kill its own child and <code>git fetch</code> becomes an orphaned process which keeps running</li> +<li>When <code>git fetch</code> eventually completes it becomes a zombie because its original parent is no longer around to wait on it</li> +</ul> +<h3 id="pid-1-process-and-waiting-on-orphans">PID 1 process and waiting on orphans<a class="zola-anchor" href="#pid-1-process-and-waiting-on-orphans" aria-label="Anchor link for: pid-1-process-and-waiting-on-orphans" + ><span class="anchor-icon">#</span></a +> +</h3> +<p>This scenario is not unique to Gitea and it is such a common pattern that safeguards have been implemented to mitigate the proliferation of zombies. Orphaned process are automatically attached to the process with PID 1, which is expected to wait on every process, whether it created them or not. When Gitea is installed from binary on GNU/Linux this is <code>/bin/init</code> and when Gitea runs from the <a href="https://github.com/go-gitea/gitea/blob/6171ea7d318c0ca8714bc6efd6a97ea4b495eb6d/Dockerfile">default docker image</a> this is <code>s6</code>: they will both wait on orphaned processes and there won't be any zombies.</p> +<h3 id="what-if-gitea-is-the-only-running-process">What if Gitea is the only running process?<a class="zola-anchor" href="#what-if-gitea-is-the-only-running-process" aria-label="Anchor link for: what-if-gitea-is-the-only-running-process" + ><span class="anchor-icon">#</span></a +> +</h3> +<p>But when Gitea runs from the <a href="https://github.com/go-gitea/gitea/blob/6171ea7d318c0ca8714bc6efd6a97ea4b495eb6d/Dockerfile.rootless">rootless docker image</a>, Gitea is the only process running in the container. Orphaned processes will have Gitea as a parent but will not wait on them and they will stay in a zombie state forever. To reproduce this problem in a minimal way:</p> +<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ docker run --name gitea -p 8080:3000 -e GITEA__security__INSTALL_LOCK=true -d gitea/gitea:1.16.8-rootless +</span><span>$ docker exec --user 1000 gitea gitea admin user create --admin --username root --password admin1234 --email root@example.com +</span></code></pre> +<p>The <code>git</code> command can then be replaced with a script that waits forever:</p> +<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ ( echo -e &#39;#!/bin/bash\nsleep infinity&#39; ) | docker exec -i --user root gitea tee /usr/bin/git +</span><span>$ docker exec --user root gitea chmod +x /usr/bin/git +</span></code></pre> +<p>Trying to create a repository from the web interface will create the conditions for a zombie to show:</p> +<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ docker exec gitea ps -o ppid,pid,comm,args +</span><span>PPID PID COMMAND COMMAND +</span><span> 0 1 gitea /usr/local/bin/gitea -c /etc/gitea/app.ini web +</span><span> 1 94 sleep [sleep] +</span><span> 1 99 sleep [sleep] +</span><span> 1 111 sleep [sleep] +</span><span> 1 164 git {git} /bin/bash /usr/bin/git -c credential.helper= -c protocol.version=2 -c uploadpack.allowfilter=true -c uploadpack.allowAnySHA1InWant=true init --bare +</span><span> 164 165 sleep sleep infinity +</span></code></pre> +<p>When the <code>git</code> process is killed by Gita, the <code>sleep</code> child will be orphaned:</p> +<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ docker exec gitea ps -o ppid,pid,comm,args +</span><span>PPID PID COMMAND COMMAND +</span><span> 0 1 gitea /usr/local/bin/gitea -c /etc/gitea/app.ini web +</span><span> 1 94 sleep [sleep] +</span><span> 1 99 sleep [sleep] +</span><span> 1 111 sleep [sleep] +</span><span> 1 165 sleep sleep infinity +</span></code></pre> +<p>Killing it will turn it into a zombie:</p> +<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ docker exec gitea kill 165 +</span><span>$ docker exec gitea ps -o ppid,pid,comm,args +</span><span>PPID PID COMMAND COMMAND +</span><span> 0 1 gitea /usr/local/bin/gitea -c /etc/gitea/app.ini web +</span><span> 1 94 sleep [sleep] +</span><span> 1 99 sleep [sleep] +</span><span> 1 111 sleep [sleep] +</span><span> 1 165 sleep [sleep] +</span></code></pre> +<h3 id="killing-a-child-process-and-all-its-children">Killing a child process and all its children<a class="zola-anchor" href="#killing-a-child-process-and-all-its-children" aria-label="Anchor link for: killing-a-child-process-and-all-its-children" + ><span class="anchor-icon">#</span></a +> +</h3> +<p>There should be no need for an admin running Gitea to worry about those gory details, it should be taken care of regardless of the environment Gitea runs in. Fortunately there is a very simple way to avoid the creation of zombies by ensuring that all Gitea child process are <a href="https://en.wikipedia.org/wiki/Process_group">process group leaders</a>. In a nutshell it means that when the child is killed all its children and grand children are also killed.</p> +<p>A <a href="https://github.com/go-gitea/gitea/pull/19865">patch was introduced in Gitea 1.17</a> and backported to Gitea 1.16.9 so that all <code>git</code> commands are created as process group leaders and solve this problem for good.</p> + + [solved] Gitea 1.15 and up: path not found or permission denied 2022-05-28T00:00:00+00:00 diff --git a/blog/index.html b/blog/index.html index c271f1c..07228f3 100644 --- a/blog/index.html +++ b/blog/index.html @@ -203,6 +203,41 @@