<contenttype="html"><p>In Gitea 1.15 the <a href="https://github.com/go-gitea/gitea/blob/cfb4c23a5009b9c236d48ac0bc156577c7d70741/custom/conf/app.example.ini">app.example.ini</a> file was changed to <a href="https://github.com/go-gitea/gitea/commit/4a84022d2559ccfc99960c7c654ee8b9b38664f7">comment out most of the values</a>. The assumption was that all values exactly matched the defaults <a href="https://github.com/go-gitea/gitea/blob/main/modules/setting/setting.go">in the source code</a>. However, there are differences, for instance for <a href="https://github.com/go-gitea/gitea/blob/cfb4c23a5009b9c236d48ac0bc156577c7d70741/modules/setting/setting.go#L771">APP_DATA_PATH</a>. Before Gitea 1.15, <code>app.example.ini</code> contained:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>APP_DATA_PATH = data
<p>and the path was relative to the <strong>directory from which the Gitea server was running</strong>. In Gitea 1.15 up to 1.16, it was commented out:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>; APP_DATA_PATH = data
<p>and the path was relative to the <strong>work path directory</strong>, as provided either via the --work-path argument or the <code>GITEA_WORK_DIR</code> environment variable. </p>
<p>When a distribution such as voidlinux <a href="https://github.com/void-linux/void-packages/blob/master/srcpkgs/gitea/patches/config.patch">uses app.example.ini</a> as a base for the Gitea package, this change indirectly creates a regression and an upgrade of Gitea <a href="https://github.com/go-gitea/gitea/issues/19367">fails with errors</a> such as <code>unable to open level db at data/data/queues/common: mkdir data: permission denied</code>. The regression did not show as soon as Gitea 1.15 became available in voidlinux because the package <a href="https://github.com/void-linux/void-packages/blob/7fc9190f0e0d557dd5031e68df4e183892d4315b/srcpkgs/gitea/patches/config.patch#L62">explicitly set <code>APP_DATA_PATH</code></a>. But this <a href="https://github.com/void-linux/void-packages/commit/19d986a2cae9ce73d32552ddb62443b5e7fa13e2">changed when Gitea 1.15.6 was packaged</a> and once the value was commented out, upgrading triggered the problem. This was worked around six month later with the <a href="https://github.com/void-linux/void-packages/commit/44b6c96fa12ce9d993c7a2ac9486d892735b7e3a">Gitea 1.16.8</a> package.</p>
<p>The <code>APP_DATA_PATH</code> directory is not the only one, the <code>[log] ROOT_PATH</code> is another example. There is an <a href="https://github.com/go-gitea/gitea/pull/19815">ongoing effort</a> to improve the situation in Gitea 1.17. With the downside of introducing breaking changes that will have an impact on all Gitea installations because the content of the <code>app.ini</code> file will be interpreted differently. In the case of <code>APP_DATA_PATH</code>, both:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>APP_DATA_PATH = data
<p>will be interpreted to be relative to the <strong>work path directory</strong>, as provided either via the --work-path argument or the <code>GITEA_WORK_DIR</code> environment variable. Every Gitea installation using <strong>APP_DATA_PATH = data</strong> will need to update the value to be an absolute path such as <strong>/var/lib/gitea/data</strong> so that it keeps pointing to the expected directory.</p>
<p>In order to prepare for the change or ensure the consistency of all path, there fortunately is a very simple <strong>solution: always use absolute paths in the <code>app.ini</code> configuration file</strong>.</p>
<contenttype="html"><p>April 12, 2022 version <a href="https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/">git v2.35.2</a> was released and addresses a security issue <a href="https://github.com/git-for-windows/git/security/advisories/GHSA-vw2c-22j4-2fh2">CVE-2022-24765</a>. It was backported to 2.30.3, v2.31.2, v2.32.1, v2.33.2, and v2.34.2 and published by distributions such as <a href="https://security-tracker.debian.org/tracker/CVE-2022-24765">Debian GNU/Linux</a>, <a href="https://www.alpinelinux.org/releases/">Alpine</a>.</p>
<p><strong>If Gitea runs as user <code>foo</code>, calls a patched Git version and a parent directory of the git repositories is owned by a user other than <code>foo</code>, it will fail</strong> with a message such as:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Failed to open repository: Git/Data Error: exit status 128 - fatal: unsafe repository (&#39;/data/git/repositories/git/data.git&#39; is owned by someone else)
<p>This started to show in the past few weeks to <a href="https://github.com/go-gitea/gitea/issues/19455">users running the Gitea binary on Windows</a> who also independently installed git v2.36. And then to people running <a href="https://github.com/go-gitea/gitea/issues/19455#issuecomment-1106331149">Gitea from snap</a>, on <a href="https://github.com/go-gitea/gitea/issues/19455#issuecomment-1106312061">a Synology NAS</a> and then people running from <a href="https://github.com/go-gitea/gitea/blob/main/Dockerfile#L2">Gitea docker images</a> which is based on <a href="https://www.alpinelinux.org/releases/">Alpine</a>.</p>
<h3 id="workarounds">Workarounds<a class="zola-anchor" href="#workarounds" aria-label="Anchor link for: workarounds"
<li>If using <a href="https://hub.docker.com/r/gitea/gitea">Gitea docker images</a>:
<ul>
<li>do not upgrade to 1.16.6 or 1.16.7, or</li>
<li>downgrade from 1.16.6 or 1.16.7 to 1.16.5 (do <strong>not</strong> downgrade from 1.17.x, it may corrupt your the Gitea database)</li>
</ul>
</li>
<li>If the Gitea binary was installed independently of git, upgrade git to a version that is <a href="https://git-scm.com/docs/git-config#Documentation/git-config.txt-safedirectory">greater or equal to 2.36</a> and disable the security check entirely with:
<ul>
<li>impersonate the <a href="https://docs.gitea.io/en-us/install-from-binary/#recommended-server-configuration">user dedicated to Gitea</a> (usually git)</li>
<p>The <a href="https://github.com/go-gitea/gitea/pull/19707">bug fix</a> is for Gitea to ensure <code>git config --global --replace-all safe.directory '*'</code> is set on its <a href="https://docs.gitea.io/en-us/install-from-binary/#recommended-server-configuration">dedicated user</a> when it initializes. It is effective on the condition that the git CLI version is <a href="https://git-scm.com/docs/git-config#Documentation/git-config.txt-safedirectory">greater or equal to 2.36</a>.</p>
<p>It is safe to <a href="https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/50/diffs">disable the security check in Gitea</a>. It is not vulnerable to <strong><a href="https://github.com/git-for-windows/git/security/advisories/GHSA-vw2c-22j4-2fh2">CVE-2022-24765</a></strong> because it calls the git CLI <a href="https://github.com/go-gitea/gitea/blob/main/modules/git/command.go#L160">after changing its working directory</a> to be the git repository targeted by the command (for instance <a href="https://github.com/go-gitea/gitea/blob/main/modules/git/diff.go#L38-L45">diff</a>) or a temporary directory. Therefore <strong>it will not explore the parent directories looking for a git configuration file</strong>.</p>
<p>The security check is triggered because the repository is owned by an unexpected user (root instead of git typically) and <strong>not because a parent directory is owned by an unexpected user</strong>. This, in itself, is a problem worth investigating but it is unrelated and was revealed by the newer security check of git even though it does not match the threat described in <strong><a href="https://github.com/git-for-windows/git/security/advisories/GHSA-vw2c-22j4-2fh2">CVE-2022-24765</a></strong>.</p>
<p>It appears non trivial to enforce a consistent ownership of files and directories, either within docker or outside docker when networked file systems are involved. The Gitea server was not troubled by this inconsistency so far because the permissions allow it to write and read where expected, regardless of the owner. It is not worth looking into but it is ancient and unrelated.</p>
<p>Gitea runs under a dedicated user, either when installed <a href="https://docs.gitea.io/en-us/install-from-binary/#recommended-server-configuration">from binary</a> or from <a href="https://docs.gitea.io/en-us/install-with-docker/">docker</a> and <a href="https://github.com/go-gitea/gitea/blob/main/modules/git/git.go#L196-L207">modifies the global git configuration</a> depending on the git version at initialization time. Fixing the problem can therefore be done by <a href="https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/50/diffs#bcd72ff867cbd1ddd5b6518c3a05b5f1a6021286_209_209">disabling the security check in the global git config file at initialization time</a>. It also requires a minimum version of git 2.36 to be installed <a href="https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/50/diffs#6651ddff6eb82c840ced7c1dddee15c6e1913dd4_44_49">in the Gitea docker image</a>. </p>
<contenttype="html"><p>The <a href="https://docs.gitea.io/en-us/upgrade-from-gitea/#upgrade-from-binary">instructions to upgrade a Gitea instance</a> only require three to four steps. They work fine most of the time but the documentation is lacking a &quot;Troubleshooting&quot; section to help out when something goes wrong. Maintaining instructions on how to diagnose and fix upgrade problems is an ambitious undertaking and requires updates every time a new case is discovered.</p>
<p>An <a href="https://forum.hostea.org/t/things-to-know-about-gitea-upgrades/39">inventory of the known upgrade issues</a> was started to figure out how to structure such a section in the documentation. The <a href="https://blog.gitea.io/">release notes</a> were analyzed all the way back to <a href="https://github.com/go-gitea/gitea/releases/tag/v1.9.6">Gitea 1.9.6</a> and the work is still in progress. Here is a sample of the tips that will be included:</p>
<ul>
<li>Upgrade directly to the latest Gitea version, there is no need to upgrade to intermediate versions.</li>
<li>If the upgrade from version x.y to version x.y+2 fails and there is a need to narrow down the problem, try upgrading to the latest minor version of each major version and verify it works.</li>
<li>etc.</li>
</ul>
<p>However, even with the best documentation, someone will eventually <strong>run into an new problem</strong> and fixing it without compromising the integrity of the data will be challenging. This is best demonstrated by a real world example that was concluded a few days ago.</p>
<h1 id="getting-help-from-the-community">Getting help from the community<a class="zola-anchor" href="#getting-help-from-the-community" aria-label="Anchor link for: getting-help-from-the-community"
<p>After <a href="https://discourse.gitea.io/t/blank-page-after-login/5051">upgrading a Gitea intsance from 1.9.6 to 1.16.5</a> the tests conducted manually did not uncover any problem. However, after going to production, some users saw a blank page after login and had to manually type the URL of the project they wanted to see in the browser. The person in charge of the upgrade never had to diagnose Gitea problem and <a href="https://discourse.gitea.io/t/blank-page-after-login/5051">reached out in the Gitea forum</a>.</p>
<blockquote>
<p><strong>Tip: explain the problem in a public forum as early as possible to get help from the community</strong></p>
</blockquote>
<p>In their post in the forum they explained how they attempted to diagnose the problem and how why they thought that only users created a few years ago were impacted. It was a detailed analysis that was concluded with a partial copy of the logs. It was unfortunately missing <a href="https://discourse.gitea.io/t/blank-page-after-login/5051/12">key information</a> that was provided only three days later. In the meantime, as they could not figure out the source of the problem, they were on the <strong>verge of <a href="https://discourse.gitea.io/t/blank-page-after-login/5051/11">accepting the loss of all the Gitea database</a> and start over from the repositories</strong>. However, once all the details were available, <a href="https://discourse.gitea.io/t/blank-page-after-login/5051/13">a workaround</a> was suggested in the forum.</p>
<blockquote>
<p><strong>Tip: focus more on providing detailed facts than exposing the attempted diagnostic</strong></p>
</blockquote>
<p>There was hope to fix Gitea and in the following days they applied the workaround. They also tried to improve it but without success and eventually accepted a <strong>partial data loss</strong> as inevitable and <a href="https://discourse.gitea.io/t/blank-page-after-login/5051/14">reported their success back to the forum</a>.</p>
<blockquote>
<p><strong>Tip: when getting support from the community, providing feedback is the best token of appreciation</strong></p>
</blockquote>
<h1 id="getting-professional-help">Getting professional help<a class="zola-anchor" href="#getting-professional-help" aria-label="Anchor link for: getting-professional-help"
<p>The <a href="https://hostea.org/gitea-clinic/">Hostea Clinic</a> is a collective of individual and companies that provides professional services to Gitea admins. They are active members of the Gitea community who <a href="https://discourse.gitea.io/u/dachary/activity">help out</a> as volunteers. They can also be hired to resolve the more complicated cases.</p>
<p>The Gitea instance that was in trouble required more than a few minutes of work and access to the database content for a proper diagnostic. They <a href="https://discourse.gitea.io/t/blank-page-after-login/5051/13">proposed their assistance</a> but although <a href="https://discourse.gitea.io/t/user-research-about-gitea-upgrade-experiences-call-for-volunteers/5063/2">well received</a>, it was not accepted.</p>
<p>When the Gitea admin explained how they chose to resolve the problem <a href="https://discourse.gitea.io/t/blank-page-after-login/5051/14">on the forum</a>, it confirmed the workaround was viable and the root problem was identified. That was enough to figure out a fix for the underlying bug with <a href="https://discourse.gitea.io/t/blank-page-after-login/5051/17">a rather simple patch</a> that was merged <a href="https://github.com/go-gitea/gitea/pull/19629">and backported</a> in the following days. But it happened too late to avoid the data loss.</p>
<p>To summarize with a timeline, here is what happened:</p>
<ul>
<li>J+1: The <strong>problem is discovered</strong> by users who see a blank page after login and a the Gitea admin tries to diagnose the problem</li>
<li>J+2: A message is sent <strong>to ask for help in the community</strong></li>
<li>J+2 to J+6: Three people in the community suggest ideas but <strong>the Gitea admin cannot figure out the root cause and is on the verge of accepting the loss of all Gitea data</strong> and restart from the git repositories</li>
<li>J+6: A <strong>workaround is suggested by the community</strong></li>
<li>J+7 to J+17: The Gitea admin applies the <strong>workaround and only looses part of the Gitea data</strong></li>
</ul>
<p>And in retrospect, here is what could have happened instead:</p>
<ul>
<li>J+1: The <strong>problem is discovered</strong> by users who see a blank page after login</li>
<li>J+1: The Gitea admin <strong><a href="https://hostea.org/gitea-clinic/">reaches out to someone at the Hostea Clinic</a></strong></li>
<li>J+2: The <a href="https://discourse.gitea.io/t/blank-page-after-login/5051/12">logs of the Gitea instance</a> are analyzed, <strong>the root cause diagnosed</strong> and <a href="https://discourse.gitea.io/t/blank-page-after-login/5051/17">a patch</a> is created to fix it.</li>
<li>J+3: If necessary a Gitea binary is created with the patch and used as a temporary replacement until the next point release is published with <a href="https://github.com/go-gitea/gitea/pull/19629">the backport</a>. The Gitea admin runs the patched Gitea binary in the meantime. <strong>There is no data loss</strong>.</li>
</ul>
<p>It does not mean all upgrade problems can be resolved so easily. But it shows, with an example, that in some cases it makes sense to get professional help.</p>
<contenttype="html"><p><em>This post was originally published on <a href="https://blog.dachary.org/2022/02/16/project-plans-for-a-hosted-gitea-online-service/">Loïc Dachary's
<h2 id="more-mature-online-services-mean-less-opportunities-to-sell-services">More mature online services mean less opportunities to sell services<a class="zola-anchor" href="#more-mature-online-services-mean-less-opportunities-to-sell-services" aria-label="Anchor link for: more-mature-online-services-mean-less-opportunities-to-sell-services"
<p>Ideally the software running an online service is rock solid and bugs
are so rare that it can run unattended. This is true of
https://wordpress.org and it is not uncommon for an instance to run for
years while upgrading themselves to get security patches. The cost of
maintaining such a service is negligible and hosting companies can offer
it for free to their customers. They make their profit by renting the
machines on which the service runs.</p>
<p>When the software is not as mature, it is more expensive to run. Bugs
need fixing, upgrades require manual intervention, users must be trained
to overcome the complexity of the user experience, etc. Well known
examples are Discourse or CiviCRM for which companies sell services to
overcome these issues.</p>
<p>But when an organization is both the author of the software and the
provider of paid services to compensate for its lack of maturity, it
creates a conflict of interest. Should they focus their effort on making
the software more mature, they would harm a business model that is based
on this very lack of maturity. For instance, if the author of a software
also sells training courses, they are not motivated to solve user
experience issues. If they did, it would lower the need for training
courses and reduce their income.</p>
<h2 id="free-software-online-services-in-the-wake-of-the-sustainability">Free Software online services in the wake of the sustainability<a class="zola-anchor" href="#free-software-online-services-in-the-wake-of-the-sustainability" aria-label="Anchor link for: free-software-online-services-in-the-wake-of-the-sustainability"