Controller and server not starting after upgrade from 4.0.3 to 4.1.2 #16842

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

I upgraded to the following images

• quay.io/argoproj/workflow-controller:v4.1.2
• quay.io/argoproj/argocli:v4.1.2

Now I get the following error from the controller and server: level=ERROR msg="failed to create initial database session: failed to create upper/db session: pq: no pg_hba.conf entry for host \"172.28.6.51\", user \"argo_test\", database \"argo_test\", no encryption (28000)"

It seems very similar to #16707 but im using postges instead of mysql.

Version(s)

v4.1.2

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

persistence: |
  archive: true
  postgresql:
    host: my-azure-postgresql-server.postgres.database.azure.com
    database: argo_test
    tableName: argo_test
    userNameSecret:
      name: argo-postgres-config
      key: user
    passwordSecret:
      name: argo-postgres-config
      key: password
    sslMode: require

Logs from the workflow controller

level=ERROR msg="failed to create initial database session: failed to create upper/db session: pq: no pg_hba.conf entry for host \"172.28.6.51\", user \"argo_test\", database \"argo_test\", no encryption (28000)"

Logs from in your workflow's wait container

-