feat: Make database migration happen automatically

[rustybee42]

• Deprecate and hide --upgrade, make it do nothing and exit
• Do the migration automatically on start
• Change some of the wording from "upgrade" to "migrate"

[iamjoemccormick]

suggestion(blocking): after thinking it over I think it would be more user friendly to just introduce a new boolean db-auto-upgrade parameter. That way we can just keep the --upgrade flag as it is today which avoids needing to update docs, training, etc.

[rustybee42]

Why? It still works as before, until we remove it in v9, which will most likely require that effort anyway due to all kind of breaking changes. And a doc/config file update is required either way as the auto update is a new feature as is the config file setting for it, regardless the name (unless we want to be silent about it, of course).

[iamjoemccormick]

Because after thinking about this further, I think we should default to db-auto-upgrade = true. If we don't default to true then users that don't read the upgrade guide are going to immediately experience an outage because the management will refuse to start, and they have to go debug why.

If users have committed to upgrading to a new version, they have also committed to upgrading the DB schema (they have to). There is no reason to ask them twice (i.e., "do you want to upgrade the mgmtd package, do you want to upgrade the mgmd db").

Since we already automatically take a backup, in the unlikely event they need to downgrade, there is a clear path. The only exception is if they let the system run for some time and make changes that affect the management database, those would need to be manually applied on the backup version. But that is unlikely, and generally if a user is downgrading support should be involved anyway.

[philippfalk]

I agree with @iamjoemccormick here. The automatic update is safe enough and should be the default in my opintion. I don't really see any downsides or failure scenarios that would be problematic here. I think we should treat simple DB schema changes as an implementation detail and just silently apply them. This way, we can avoid extra steps during the upgrade and give the user a smoother upgrade experience.

I also think preserving the original explicit --upgrade behavior for compatibility reasons makes sense. I would probably actually go even further and question whether we want to make automatic upgrades opt-out at all. The mgmtd will never work on an old schema anyway, so what is the point of that configuration?

[rustybee42]

This discussion is perpendicular to the original suggestion to use a commondb_upgrade or upgrade and db_auto_upgrade (both can be implemented both ways).

That said, if we do that, then I tend to agree with

I would probably actually go even further and question whether we want to make automatic upgrades opt-out at all. The mgmtd will never work on an old schema anyway, so what is the point of that configuration?

and would instead just remove the --upgrade completely and just let it always upgrade automatically.

[iamjoemccormick]

Yeah they are technically separate discussions, but also kind of related because it sounds like the conclusion we came to is to just remove the --upgrade flag completely and always auto-upgrade the database.

I propose we don't actually remove it, just hide it and print a deprecation warning if it is used. There might be user scripts that already try to implement auto-upgrade behavior.

[rustybee42]

Changed as discussed. Note that I force pushed intentionally as the new changes are completely different, the previous review is obsolete now.

[iamjoemccormick]

suggestion(non-blocking): This would try to upgrade the DB on any error from the schema_check() including the outside the valid range error where this binary shouldn't try to migrate the on-disk DB.

The migration would never actually be attempted due to the second check in migrate_schema(), but it would try to take a backup then later fail. It would be better just to fail fast.