Hello,
We have a strange issue with stolon deployed on k8s. When we shut down 1 of 3 worker server and start it again components of stolon (sentinel, keeper) that was assigned to this worker have an error.
keeper logs:
2021-03-30T15:56:22.688Z INFO cmd/keeper.go:2007 exclusive lock on data dir taken
2021-03-30T15:56:22.717Z INFO cmd/keeper.go:521 keeper uid {“uid”: “keeper2”}
2021-03-30T15:56:22.824Z INFO cmd/keeper.go:1030 no db assigned
2021-03-30T15:56:22.851Z ERROR cmd/keeper.go:669 cannot get configured pg parameters {“error”: “dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory”}
2021-03-30T15:56:23.253Z ERROR cmd/keeper.go:820 failed to update keeper info {“error”: “update failed: Pod “pg-fdatabase-keeper-2” is invalid: spec: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds or spec.tolerations (only additions to existing tolerations)\n core.PodSpec{\n \tVolumes: []core.Volume{{Name: “data”, VolumeSource: core.VolumeSource{PersistentVolumeClaim: &core.PersistentVolumeClaimVolumeSource{ClaimName: “data-pg-database-keeper-2”}}}, {Name: “config”, VolumeSource: core.VolumeSource{ConfigMap: &core.ConfigMapVolumeSource{LocalObjectReference: core.LocalObjectReference{Name: “pg-database”}, DefaultMode: &420}}}, {Name: “stolon-secrets”, VolumeSource: core.VolumeSource{Secret: &core.SecretVolumeSource{SecretName: “pg-database”, DefaultMode: &420}}}, {Name: “pg-database-token-4llqj”, VolumeSource: core.VolumeSource{Secret: &core.SecretVolumeSource{SecretName: “pg-database-token-4llqj”, DefaultMode: &420}}}},\n \tInitContainers: nil,\n \tContainers: []core.Container{\n \t\t{\n \t\t\t… // 15 identical fields\n \t\t\tTerminationMessagePath: “/dev/termination-log”,\n \t\t\tTerminationMessagePolicy: “File”,\n- \t\t\tImagePullPolicy: “Always”,\n+ \t\t\tImagePullPolicy: “IfNotPresent”,\n \t\t\tSecurityContext: nil,\n \t\t\tStdin: false,\n \t\t\t… // 2 identical fields\n \t\t},\n \t},\n \tEphemeralContainers: nil,\n \tRestartPolicy: “Always”,\n \t… // 24 identical fields\n }\n”}
Stolon config:
1 master
2 slave
synchronous replications - true for one standby
Have someone have an idea what can make this issue?