8.2 KiB
+++ title = "[tutorial] A gentle introduction to the gitea doctor" date = 2022-06-14 description = "The doctor command is useful to check the health of a running Gitea instance. Specially after performing an upgrade." [taxonomies] tags = ['hostea', 'gitea', 'troubleshoot', 'problem', 'tutorial']
[extra] author = 'dachary' +++
While helping people with their upgrades in the Gitea forum or at the Hostea clinic, I realized that few Gitea admins know about the gitea doctor command and decided to write this blog post as a gentle introduction.
An apple a day keeps the doctor away
Or in our case, Gitea versions below 1.11.5. Since then, the gitea doctor is available and is designed to run against a specific Gitea version. It would not be a good idea to try to run the doctor from Gitea 1.16 to verify the sanity of a Gitea 1.2 instance: it will be confused by how the database is organized and a number of other details. Historical fun fact: the gitea doctor was backported to Gitea 1.10.5 and Gitea 1.10.6 and may be of help if you run this particular version and are facing the problem that motivated the backport.
With each version gitea doctor improves and gains new capabilities. For instance, in Gitea 1.17 it becomes aware of orphaned pull requests and is able to fix them. If such a problem exists in Gitea 1.16, it does not know about it.
Calling the doctor
In the following, examples are based on a Gitea 1.16.8 instance you can run as follows:
$ docker run --name gitea -p 3000:3000 -e GITEA__security__INSTALL_LOCK=true -d gitea/gitea:1.16.8-rootless
$ docker exec gitea gitea admin user create --admin --username root --password admin1234 --email root@example.com
$ docker exec gitea mkdir /var/lib/gitea/data/log
And then you can go to the web interface to create a test repository, with an initial README.md file. When this is done the doctor can be called as follows:
$ docker exec gitea gitea doctor --all
[1] Check paths and basic configuration
- [I] Configuration File Path: "/etc/gitea/app.ini"
- [I] Repository Root Path: "/var/lib/gitea/git/repositories"
- [I] Data Root Path: "/var/lib/gitea"
- [I] Custom File Root Path: "/var/lib/gitea/custom"
- [I] Work directory: "/var/lib/gitea"
- [I] Log Root Path: "/var/lib/gitea/data/log"
OK
[2] Check if there is garbage storage files
OK
[3] Check Database Version
OK
[4] Check consistency of database
OK
[5] Check if user with wrong type exist
OK
[6] Check if OpenSSH authorized_keys file is up-to-date
OK
[7] Check if SCRIPT_TYPE is available
- [I] ScriptType bash is on the current PATH at /bin/bash
OK
[8] Check if hook files are up-to-date and executable
OK
[9] Recalculate Stars number for all user
OK
[10] Check old archives
- [I] 0 old archives in repository need to be deleted
OK
[11] Enable push options
- [I] Checked 1 repositories, 0 need updates.
OK
[12] Check for incorrectly dumped repo_units (See #16961)
- [W] Found 0 broken repo_units
OK
[13] Recalculate merge bases
- [W] 0 PRs with incorrect mergebases of 0 PRs total in 1 repos
OK
[14] Check git-daemon-export-ok files
- [I] Checked 1 repositories, 0 need updates.
What does the doctor know?
Although the doctor can be compared to fsck(8), it does not know everything. It took decades for fsck to become the ultimate authority on finding problems on file systems and reliably fixing them without losing data. Nowadays, only a handful of people in the world are brave enough to manually attempt a file system recovery when fsck cannot recover from a data loss.
The first doctor version is two years old and Gitea admins are still routinely running SQL queries against the database or moving files around when trying to figure out why a Gitea instance is not behaving as it should. It is however worth checking if the doctor does not already have a solution by listing all it can do:
$ docker exec gitea gitea doctor --list
Default Name Title
* paths Check paths and basic configuration
storages Check if there is garbage storage files
* check-db-version Check Database Version
check-db-consistency Check consistency of database
* check-user-type Check if user with wrong type exist
* authorized-keys Check if OpenSSH authorized_keys file is up-to-date
script-type Check if SCRIPT_TYPE is available
hooks Check if hook files are up-to-date and executable
recalculate-stars-number Recalculate Stars number for all user
check-old-archives Check old archives
enable-push-options Enable push options
fix-broken-repo-units Check for incorrectly dumped repo_units (See #16961)
recalculate-merge-bases Recalculate merge bases
check-git-daemon-export-ok Check git-daemon-export-ok files
And then call the check that looks interesting:
$ docker exec gitea gitea doctor --run authorized-keys
[1] Check if OpenSSH authorized_keys file is up-to-date
OK
The challenge is to figure out which check does what and at the moment the best source of information is ... the sources themselves. The doctor.go command is the entry point and the doctor directory contains the rest.
Some checks are straightforward to understand, even if you do not know Go, such as the authorized-keys check. Others are much more involved and your best chance is to ask the Gitea chatroom for help.
Is it going to hurt?
By default the doctor (very much like fsck -N) only performs non destructive checks and displays diagnostics, with an indication of how serious the problem is. In the example above, there only are lines with [I] (which indicates an information) and [W] which indicates a warning that can be ignored but may be worth looking into. Those two warnings are actually just informational and should be labelled as [I], which has been fixed in a more recent version of the doctor.
Now let's do something bad: remove the permissions from a hook in our repository:
$ docker exec gitea chmod -x /var/lib/gitea/git/repositories/root/test.git/hooks/post-receive
Run the doctor with the check supposed to find that out:
$ docker exec gitea gitea doctor --run hooks
[1] Check if hook files are up-to-date and executable
- [W] old hook file /var/lib/gitea/git/repositories/root/test.git/hooks/post-receive is not executable
Ask it to fix this with the --fix flag:
$ docker exec gitea gitea doctor --run hooks --fix
[1] Check if hook files are up-to-date and executable
- [W] Regenerated hooks for root/test
- [W] old hook file /var/lib/gitea/git/repositories/root/test.git/hooks/post-receive is not executable
And run it one last time to check all is well:
$ docker exec gitea gitea doctor --run hooks
[1] Check if hook files are up-to-date and executable
OK
Even when the doctor is unable to fix a problem, it can help by showing extensive debug output which can be found, by default, in the doctor.log file in the directory from which it runs. Or it can be displayed on the standard output with --log-file -, which is most convenient when running in docker.
Going further
If that was helpful to you, I would very much appreciate if you send me a message on Mastodon. It will encourage me to write more blog posts to share what I learn about Gitea. Even better: you could send a pull request to improve the doctor and help it mature.