I was trying to upgrade Grafana to v11 (which finally supports nested folder after seven years). The update (replacing container) worked until Grafana tried to connect to a database, which failed with the following message:
logger=sqlstore t=2024-07-23T11:55:21.018796684Z level=info msg="Connecting to DB" dbtype=mysql
logger=migrator t=2024-07-23T11:55:21.027490163Z level=info msg="Locking database"
logger=migrator t=2024-07-23T11:55:21.030442889Z level=error msg="Failed to lock database" error=EOF
I checked the user’s permissions, which were OK; therefore, the problem should not be a result of bad permissions:
GRANT ALL PRIVILEGES ON `grafana-dev`.* TO `grafana-dev`@
Increasing the log level also did not show anything new; therefore, the problem really have to be in the database.
[log.console]
level = trace
I am not an expert on MySQL clusters so after discussion with a colleague it was clear that the problem was activated strict mode on our Percona XtraDB cluster we use for production. It turned out that Grafana locks the database during migrations since v11 by default - which turns out to be a problematic for the strict mode when in ENFORCING mode. Although database lock is a nice feature (introduced some time ago) that prevents concurrent migrations, it is unfortunate feature if you use the strict mode. Fortunatelly it can be disabled in the configuration file (and possibly should not bring any issues, because we are running single instance Grafana with dedicated mysql user and database):
[database]
migration_locking = false
If for some reason you want play with the strict mode itself, set the strict mode to PERMISSIVE, upgrade the Grafana, and switch the strict mode back to ENFORCING (this can be done during runtime, so use the following command as the mysql root user). But remember that conteinerized Grafana runs the migration every time it is started.
mysql> SET GLOBAL pxc_strict_mode=PERMISSIVE;
mysql> SET GLOBAL pxc_strict_mode=ENFORCING;