What happened:
When changing passwords for database users in stolon (postgres), both keeper and sentinel logs start to print errors, examples of errors below. (Current DB username is stolon.)
Keeper logs:
2020-03-12T12:02:15.361Z ERROR cmd/keeper.go:650 cannot get configured pg parameters {“error”: “pq: password authentication failed for user “stolon””}
–
LOG: incomplete startup packet
FATAL: password authentication failed for user “stolon”
DETAIL: Password does not match for user “stolon”.
Connection matched pg_hba.conf line 1: “local postgres stolon md5”
2020-03-12T12:02:18.949Z ERROR cmd/keeper.go:936 failed to get replication slots {“error”: “pq: password authentication failed for user “stolon””}
2020-03-12T12:02:18.949Z ERROR cmd/keeper.go:1427 error updating replication slots {“error”: “pq: password authentication failed for user “stolon””}
FATAL: password authentication failed for user “stolon”
DETAIL: Password does not match for user “stolon”.
Sentinel logs:
2020-03-12T12:01:51.498Z INFO cmd/sentinel.go:980 master db is failed {“db”: “cac72350”, “keeper”: “keeper1”}
–
2020-03-12T12:01:51.498Z INFO cmd/sentinel.go:991 trying to find a new master to replace failed master
2020-03-12T12:01:51.499Z ERROR cmd/sentinel.go:994 no eligible masters
What you expected to happen:
No those errors or warning logs appear.
How to reproduce it (as minimally and precisely as possible):
Login to PostgreSQL console “psql” to change the password, examples as below. Then the error log occurs.
sh-4.2$ psql -h postgresql-ha-stolon-proxy -p 5432 -U stolon postgres
Password for user stolon:
psql (9.6.14)
Type “help” for help.
postgres=# \password
Enter new password:
Enter it again:
postgres=#
postgres=# \q
sh-4.2$
Anything else we need to know?:
Actually the new password can work, although there’s the error printing.
Also, check the stolon cluster status and it’s fine.
sh-4.2# stolonctl --cluster-name=stolon-cluster --store-backend=etcdv3 --store-endpoints=http://etcd1:2379,http://etcd2:2379,http://etcd3:2379 status
=== Active sentinels ===
ID LEADER
2d410826 true
c8c0353a false
=== Active proxies ===
ID
29acec18
e4affb2c
=== Keepers ===
UID HEALTHY PG LISTENADDRESS PG HEALTHY PG WANTEDGENERATION PG CURRENTGENERATION
keeper0 true 10.131.2.89:5432 false 2 2
keeper1 true 10.129.2.146:5432 false 3 3
=== Cluster Info ===
Master: keeper1
===== Keepers/DB tree =====
keeper1 (master)
└─keeper0
Environment:
OKD 3.9
- Stolon version:
sorintlab-stolon:0.12.0 - Stolon running environment (if useful to understand the bug):
OKD 3.9