Hostea
/
admin-guide
3
1
Fork 1
Code Issues Pull Requests Releases Activity

upgrade to forgejo

master
Loïc Dachary 2023-01-08 15:46:23 +01:00
parent e36bd26e40
commit 1837860563
Signed by: dachary
GPG Key ID: 992D23B392F9E4F2
1 changed files with 27 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

49
README.md
View File

@ -1,39 +1,39 @@
This guide helps Gitea admins perform upgrades safely and provides guidance to troubleshoot problems. It covers [Gitea 1.2.0](https://github.com/go-gitea/gitea/releases/tag/v1.2.0) and above.
This guide helps Forgejo admins perform upgrades safely and provides guidance to troubleshoot problems. It covers [Gitea 1.2.0](https://github.com/go-gitea/gitea/releases/tag/v1.2.0) and above.
There are [many ways to install Gitea](https://docs.gitea.io/en-us/install-with-docker/) on various Operating Systems, each with their particular challenges and potential for trouble. The scope of this guide is limited to what [Gna!](https://gna.org) uses daily:
* Docker based installation (either [root](https://docs.gitea.io/en-us/install-with-docker/) or [rootless](https://docs.gitea.io/en-us/install-with-docker-rootless/)) using the [Gitea images published on the Docker hub](https://hub.docker.com/r/gitea/gitea).
* Docker based installation (either [root or rootless](https://forgejo.org/download/) using the [Forgejo images](https://codeberg.org/forgejo/-/packages/container/forgejo/versions).
* Any GNU/Linux based Operating Systems
It is the source of truth that Gna! members use when maintaining the infrastructure and developing the service.
#### Backup
In all cases the only reliable way to perform a backup is with a [synchronized point-in-time snapshot of all the storage used by Gitea](https://github.com/go-gitea/gitea/issues/19752#issuecomment-1133627365). For instance, if everything (SQLite database + repositories etc.) is on a single QCOW2 disk attached to a virtual machine, a [qemu snapshot](https://wiki.qemu.org/Features/Snapshots) guarantees the backup is consistent.
In all cases the only reliable way to perform a backup is with a [synchronized point-in-time snapshot of all the storage used by Forgejo](https://github.com/go-gitea/gitea/issues/19752#issuecomment-1133627365). For instance, if everything (SQLite database + repositories etc.) is on a single QCOW2 disk attached to a virtual machine, a [qemu snapshot](https://wiki.qemu.org/Features/Snapshots) guarantees the backup is consistent.
If Gitea uses a S3 storage for attachments with a PostgresQL database, stores Git repositories on a remote file system and queues in a Redis server, the only way to ensure a consistent state is to shutdown Gitea during the backup.
If Forgejo uses a S3 storage for attachments with a PostgresQL database, stores Git repositories on a remote file system and queues in a Redis server, the only way to ensure a consistent state is to shutdown Forgejo during the backup.
In the simplest case where everything is on a single file system and if the instance is not busy (no mirrors, no users), the backup can be done with:
* `gitea dump` to collect everything into one zip file
* `psql/mysql/sqlite dump`. Although the zip file created by `gitea dump` contains a copy of the database it (i) does not work across Gitea versions (it does not have the version table) and (ii) has serious long standing open bugs that may introduce problems when re-injecting the SQL dump in a new database (such as [this one](https://github.com/go-gitea/gitea/issues/8034) or [this one](https://github.com/go-gitea/gitea/issues/19752) or [that one](https://github.com/go-gitea/gitea/issues/12470) or [that one](https://matrix.to/#/!bXIgKPtRdSXHBnVBaz:osgeo.org/$dKg_hMbfmY3lKbnBwsqvlOAwmqfNBm4FR-jcUpBKL5w?via=libera.chat&via=t2bot.io&via=matrix.org)).
* `psql/mysql/sqlite dump`. Although the zip file created by `gitea dump` contains a copy of the database it (i) does not work across Forgejo versions (it does not have the version table) and (ii) has serious long standing open bugs that may introduce problems when re-injecting the SQL dump in a new database (such as [this one](https://github.com/go-gitea/gitea/issues/8034) or [this one](https://github.com/go-gitea/gitea/issues/19752) or [that one](https://github.com/go-gitea/gitea/issues/12470) or [that one](https://matrix.to/#/!bXIgKPtRdSXHBnVBaz:osgeo.org/$dKg_hMbfmY3lKbnBwsqvlOAwmqfNBm4FR-jcUpBKL5w?via=libera.chat&via=t2bot.io&via=matrix.org)).
#### Verify Gitea works
#### Verify Forgejo works
It is **critical** to verify that Gitea works very carefully. Restoring the backup done before the upgrade is easy and does not loose any information. But if a problem is discovered days or weeks after the upgrade, it will not be an option and fixing it on a live Gitea instance will be much more challenging.
It is **critical** to verify that Forgejo works very carefully. Restoring the backup done before the upgrade is easy and does not loose any information. But if a problem is discovered days or weeks after the upgrade, it will not be an option and fixing it on a live Forgejo instance will be much more challenging.
* Run `gitea doctor --all --log-file /tmp/doctor.log` and make sure it does not report any problem. The `gitea doctor` command is available for [Gitea 1.10.5](https://github.com/go-gitea/gitea/blob/v1.10.5/cmd/doctor.go), Gitea 1.10.6 and [Gitea >= 1.11.5](https://github.com/go-gitea/gitea/blob/v1.11.5/cmd/doctor.go).
* Manually verify via the web interface. Making a checklist of a typical use case is a good way to not miss anything.
* If there is a problem of any kind, [increase the log level](https://docs.gitea.io/en-us/logging-configuration/#debugging-problems) and take a look at the logs. If you cannot figure out what the problem is, ask for help [in the Gna! forum](https://forum.gna.org/c/clinic/5) before trying to fix it.
#### Gitea preparation
#### Forgejo preparation
* Manually analyze (reading the patches to the sources of the template directory) and update the [customized CSS / content](https://docs.gitea.io/en-us/upgrade-from-gitea/#take-care-about-customized-templates).
* [Do not use `gitea help`](https://discourse.gitea.io/t/gitea-custom-themes-not-working/4040/7) to figure out the location of `CustomPath`, look at the configuration tab of the administration panel when logged in as an admin.
* `gitea manager flush-queues`. If it timesout, run it again with a more generous `--timeout` argument. It is important because the queues contain serialized data that is not guaranteed to be backward compatible between versions. See [this breaking change example](https://blog.gitea.io/2022/03/gitea-1.16.4-is-released/#breaking-change-refactor-mirror-code--fix-starttomirror-19075httpsgithubcomgo-giteagiteapull19075) to better understand why this is necessary.
* With [Gitea version >=1.17.3](https://github.com/go-gitea/gitea/pull/15928)
* With [Forgejo >= 1.18.0-1 and Gitea version >=1.17.3](https://github.com/go-gitea/gitea/pull/15928)
* go to the web admin panel and pause all queues
* With Gitea version < 1.17.3
* Shutdown Gitea so that no new entry is created in the queues
* Shutdown Forgejo so that no new entry is created in the queues
---
* **Pending question:**
@ -41,7 +41,7 @@ It is **critical** to verify that Gitea works very carefully. Restoring the back
#### Docker version
* Gitea >= 1.17 requires [docker >= 20.10.6](https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0) otherwise [mysterious](https://github.com/go-gitea/gitea/issues/20162) [problems](https://github.com/go-gitea/gitea/issues/20103) will happen (mysterious in the sense that the problem will manifest itself with a problem different from an error message reading "Docker must be upgraded").
* Forgejo >= 1.18 and Gitea >= 1.17 requires [docker >= 20.10.6](https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0) otherwise [mysterious](https://github.com/go-gitea/gitea/issues/20162) [problems](https://github.com/go-gitea/gitea/issues/20103) will happen (mysterious in the sense that the problem will manifest itself with a problem different from an error message reading "Docker must be upgraded").
#### Upgrade and verify
@ -51,23 +51,28 @@ It is **critical** to verify that Gitea works very carefully. Restoring the back
* Upgrade to 1.5.3 and manually verify it runs
* Upgrade to 1.6.4 and manually verify it runs
* If the upgrade is from a gitea version greater or equal to 1.6.4 that is not mentioned to be problematic below, upgrade directly to the latest stable Gitea version, there is no need to upgrade to intermediate versions.
* Upgrade using the full version number (for instance 1.16.9) and avoid using versions such as "latest" or versions without the minor release number such as 1.17. In theory these shorter version names are associated with a version according to a certain logic: for instance "latest" is supposed to be the latest stable version. But there has been cases in the past where this logic was not implemented correctly and caused unexpected upgrades ([1.18.0-rc0 tagged as latest](https://pouet.chapril.org/@gna/109252267536406833), [1.17.0-rc1 tagged as latest](https://mastodon.online/@hostea/108514134565401798) and 1.16.0-rc1 tagged as latest).
* Upgrade using the full version number (for instance 1.16.9) and avoid using versions such as "latest" or versions without the minor release number such as 1.18. In theory these shorter version names are associated with a version according to a certain logic: for instance "latest" is supposed to be the latest stable version. But there has been cases in the past where this logic was not implemented correctly and caused unexpected upgrades ([1.18.0-rc0 tagged as latest](https://pouet.chapril.org/@gna/109252267536406833), [1.17.0-rc1 tagged as latest](https://mastodon.online/@hostea/108514134565401798) and 1.16.0-rc1 tagged as latest).
* [Verify Gitea works](#verify-gitea-works-2)
#### Troubleshoot
* [Increase the log level](https://docs.gitea.io/en-us/logging-configuration/#debugging-problems). If the Gitea version is recent they can be modified dynamically with `gitea manager logging add console -n traceconsole -l TRACE -e '((modules/git)|(services/mirror))'`
* If the upgrade from version x.y to version x.y+2 fails and there is a need to narrow down the problem, try upgrading to the latest minor version of each major version and verify it works. It will show which major version causes the issue and help debug the problem. For instance, if upgrading from 1.14.5 to 1.17.3 does not work:
* Upgrade from 1.14.5 to 1.14.6 (the last minor version of the 1.14 major version) and [verify Gitea works](#verify-gitea-works-2).
* Upgrade to 1.15.11 (the last minor version of the 1.15 major version) and [verify Gitea works](#verify-gitea-works-2).
* Upgrade to 1.17.3 (the last minor version of the 1.17 major version at this point in time) and [verify Gitea works](#verify-gitea-works-2).
* [Increase the log level](https://docs.gitea.io/en-us/logging-configuration/#debugging-problems). If the Forgejo version is recent they can be modified dynamically with `gitea manager logging add console -n traceconsole -l TRACE -e '((modules/git)|(services/mirror))'`
* If the upgrade from version x.y to version x.y+2 fails and there is a need to narrow down the problem, try upgrading to the latest minor version of each major version and verify it works. It will show which major version causes the issue and help debug the problem. For instance, if upgrading from Gitea 1.14.5 to Forgejo 1.18.0-1 does not work:
* Upgrade from Gitea 1.14.5 to Gitea 1.14.6 (the last minor version of the 1.14 major version) and [verify Forgejo works](#verify-forgejo-works-2).
* Upgrade to Gitea 1.15.11 (the last minor version of the 1.15 major version) and [verify Forgejo works](#verify-forgejo-works-2).
* Upgrade to Forgejo 1.18.0-1 (the last minor version of the 1.18 major version at this point in time) and [verify Forgejo works](#verify-forgejo-works-2).
### List of latest major versions
The database version is stored in the database and can be retrieved with **select * from version**. If the version found in the database does not match what is in the following table, it probably means the instance was installed from the development tree instead of a package and **the database may be in a state that has not been tested for upgrades**.
| Forgejo version | Date | Database |
| ----------------| ---- | -------- |
|[1.18.0-1](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#1-18-0-1) | December 2022 | v231 |
| Gitea version | Date | Database |
| --------------| ---- | ---------------- |
| --------------| ---- | -------- |
|[1.17.4](https://github.com/go-gitea/gitea/releases/tag/v1.17.4) | December 2022 | v224 |
|[1.17.3](https://github.com/go-gitea/gitea/releases/tag/v1.17.3) | October 2022 | v224 |
|[1.16.9](https://github.com/go-gitea/gitea/releases/tag/v1.16.9) | July 2022 | v211 |
|[1.15.11](https://github.com/go-gitea/gitea/releases/tag/v1.15.11) | January 2022| v189 |
@ -80,7 +85,7 @@ The database version is stored in the database and can be retrieved with **selec
### When upgrading from a specific version...
* Any version before [1.17](https://github.com/go-gitea/gitea/releases/tag/v1.17.3)
* Any version before [Gitea 1.17](https://github.com/go-gitea/gitea/releases/tag/v1.17.4)
* preserve a custom gitconfig: [episode 1](https://gna.org/blog/1-17-breaking-episode-1/), [episode 2](https://gna.org/blog/1-17-breaking-episode-2/)
* [1.13.0](https://blog.gitea.io/2020/12/gitea-1.13.0-is-released/)
* The Webhook shared secret inside the webhook payload has been deprecated and will be removed in 1.14.0: https://github.com/go-gitea/gitea/issues/11755 please use the secret header that uses an hmac signature to validate the webhook payload.
@ -102,7 +107,7 @@ In your config, you can check the [security](https://docs.gitea.io/en-us/config-
* All versions
* Symptom: [blank / 500 page after login when running SQLite](https://github.com/go-gitea/gitea/issues/18650)
* Workaround: upgrade to [1.17.3 or greater](https://github.com/go-gitea/gitea/pull/19656) and run `gitea doctor --all --fix`
* Workaround: upgrade to [Forgejo 1.18.0-1 or greater](https://github.com/go-gitea/gitea/pull/19656) and run `gitea doctor --all --fix`
* [1.12.x to 1.16.x](https://discourse.gitea.io/t/fixing-broken-avatar-icons-after-upgrade/4909)
* Workaround: [manually delete broken avatars](https://discourse.gitea.io/t/fixing-broken-avatar-icons-after-upgrade/4909/2)
@ -112,7 +117,7 @@ In your config, you can check the [security](https://docs.gitea.io/en-us/config-
### Versions with known issues
Gogs from before September 2015 migrated to Gitea 1.17 may have a [dangling `pull_repo` table](https://forum.gna.org/t/gitea-upgrade-from-gogs-to-1-16-8-unmigrated-pull-repo-table/73) and the corresponding pull requests will be removed by `gitea doctor --fix --all`.
Gogs from before September 2015 migrated to Forgejo 1.18 may have a [dangling `pull_repo` table](https://forum.gna.org/t/73) and the corresponding pull requests will be removed by `gitea doctor --fix --all`.
[From the 1.11.3 release notes](https://blog.gitea.io/2020/03/gitea-1.11.3-and-1.10.6-released/):
@ -131,4 +136,4 @@ We remind users that a bug was discovered with `gitea dump` in 1.14.3–1.14.6 a
* [Autoregistration via email not working](https://github.com/go-gitea/gitea/pull/19261) (introduced in [1.16.2](https://github.com/go-gitea/gitea/pull/18804) fixed in [1.16.6](https://github.com/go-gitea/gitea/pull/19312))
* `git push` fails after [git upgrades to v2.35.2](https://discourse.gitea.io/t/upgrading-git-push-fails-due-to-changed-security-policy-safe-directory/5185) or [Gitea docker image >= 1.16.6](https://discourse.gitea.io/t/upgrading-git-push-fails-due-to-changed-security-policy-safe-directory/5185)
* `app.example.ini` changed in a non backward compatible way [in Gitea 1.15 and up and causing path not found or permission denied](https://gna.org/blog/path-not-found/).
* [unable to login into accounts](https://github.com/go-gitea/gitea/issues/19897) created with an email valid prior to 1.16.4 but [invalid for versions >=1.16.4](https://github.com/go-gitea/gitea/pull/19085). As of gitea >=1.17 `gitea doctor` [shows which emails are subject to this regression](https://github.com/go-gitea/gitea/pull/19903)
* [unable to login into accounts](https://github.com/go-gitea/gitea/issues/19897) created with an email valid prior to 1.16.4 but [invalid for versions >=1.16.4](https://github.com/go-gitea/gitea/pull/19085). As of Gitea >=1.17 `gitea doctor` [shows which emails are subject to this regression](https://github.com/go-gitea/gitea/pull/19903)