Gitlab failed upgrade postgresql 12

I run my gitlab with podman. This has been working good. I recently ran in to problem with upgrade to 13.8. With this version postresql default version is 12.x.

I analyzed why podman image did not start, and found out that it failed on upgrade of postgresql from 11 to 12. Message was kind of cryptic:

Creating temporary data directory:Error creating new directory: /var/opt/gitlab/postgresql/data.12

First I was thinking it was permission, that was wrong. I tried to premake the directory, same error. This did not make any sense. After some internet search I found a work around to skip upgrade. This can be done by starting container with:

-e GITLAB_SKIP_PG_UPGRADE=true

With this I was able to start the container, but database did not get upgraded. I attached to the container with:

podman exec -ti gitlab /bin/bash

And tried to upgrade database manual with:

gitlab-ctl pg-upgrade -w

I got same error as before, but this time I looked closer. And see following:

STDERR: su: System error

Looks like the container do not accept to change user. When i tried to change user with sudo I got following message:

sudo: unable to send audit message: Operation not permitted

So actually it is auditing of change user that fail. This is probably something that need to be fixed in the gitlab-ce image. For now you can get around it by allow auditing by run container with:

--cap-add audit_write

I added this to my podman run command and removed -e GITLAB_SKIP_PG_UPGRADE=true and now when I run the container it upgrades the database how it is supposed to and all is good again.