Keepers on VMs and the rest in Kubernetes

General Discussions
#1

Hi, I want to deploy it in a way that postgre is running on vms but the rest of stolon components running inside kubernetes. Is that possible?
Because it is easy to automatically install in kubernetes and I didn’t find up to date ansible roles to do everything on VMs. I need postgre to be running on VMs in order to have easy access to disks, os settings and easier troubleshooting in prod.

#2

Hi. I think - yes. It is possible. Yesterday I started stolon on three VMs without kubernetes and swarm. Unfortunately stolon start to work incorrectly in swarm since docker 20.* version. That is why I am trying to start stolon without swarm. I started it using three VM nodes and docker-compose. Cannot provide full ansible playbook but here is a points:

  1. start etcd in network_mode=host (did not work with poblished ports and I did not investigate it)
  2. Do stolon init as usual
  3. start other components
  • keeper components should also use network mode host
  • keeper components shouls use different port then 5432
  • 5432 used by proxy component and proxy need accessible destination where to proxy connection

Here is few docker-compose templates for ansible:

version: "3.7"
services:
  etcd:
    image: quay.io/coreos/etcd:{{ stolon_etcd_version }}
    container_name: etcd
    network_mode: host
    hostname: {{ inventory_hostname }}
    restart: always
    extra_hosts:
{% for i in groups['stolon'] %}
      - "{{ i }}:{{ hostvars[i].ansible_host }}"
{% endfor %}
    command:
      - etcd
      - --name={{ inventory_hostname }}
      - --data-dir=data.etcd
      - --advertise-client-urls=http://{{ inventory_hostname }}:2379
      - --listen-client-urls=http://0.0.0.0:2379
      - --initial-advertise-peer-urls=http://{{ inventory_hostname }}:2380
      - --listen-peer-urls=http://0.0.0.0:2380
      - --initial-cluster={% for i in groups['stolon'] %}{{ i }}=http://{{ i }}:2380{{ "," if not loop.last else "\n" }}{% endfor %}
      - --initial-cluster-state=new
      - --initial-cluster-token={{ stolon_etcd_token }}
      - --log-level=warn
      - --auto-compaction-mode=revision
      - --auto-compaction-retention=20
      - --host-whitelist=127.0.0.1,{% for i in groups['stolon'] %}{{ hostvars[i].ansible_host }}{{ "," if not loop.last else "\n" }}{% endfor %}
    volumes:
      - etcd:/data.etcd

volumes:
  etcd:
    driver: local

version: "3.7"
services:
  sentinel:
    image: sorintlab/stolon:master-pg13
    extra_hosts:
{% for i in groups['stolon'] %}
      - "{{ i }}:{{ hostvars[i].ansible_host }}"
{% endfor %}
    container_name: stolon-sentinel
    restart: always
    command:
      - gosu
      - stolon
      - stolon-sentinel
      - --cluster-name={{ stolon_cluster_name }}
      - --store-backend=etcdv3
      - --store-endpoints={% for i in groups['stolon'] %}http://{{ i }}:2379{{ "," if not loop.last else "\n" }}{% endfor %}
#     - --metrics-listen-address=0.0.0.0:8080
      - --log-level=info

  keeper:
    image: sorintlab/stolon:master-pg13
    hostname: {{ inventory_hostname }}
    extra_hosts:
{% for i in groups['stolon'] %}
      - "{{ i }}:{{ hostvars[i].ansible_host }}"
{% endfor %}
    container_name: stolon-keeper
    network_mode: host
    restart: always
    environment:
      - PGDATA=/var/lib/postgresql/data
    volumes:
      - pgkeeper:/var/lib/postgresql/data
    command:
      - gosu
      - stolon
      - stolon-keeper
      - --pg-listen-address={{ ansible_host }}
      - --pg-repl-username=replication
      - --uid={{ inventory_hostname }}
      - --pg-su-username=postgres
      - --pg-su-password={{ stolon_su_password }}
      - --pg-repl-password={{ stolon_repl_password }}
      - --data-dir=/var/lib/postgresql/data
      - --cluster-name={{ stolon_cluster_name }}
      - --store-backend=etcdv3
      - --store-endpoints={% for i in groups['stolon'] %}http://{{ i }}:2379{{ "," if not loop.last else "\n" }}{% endfor %}
#     - --metrics-listen-address=0.0.0.0:8080
      - --pg-port=5433
      - --log-level=info

  proxy:
    image: sorintlab/stolon:master-pg13
    extra_hosts:
{% for i in groups['stolon'] %}
      - "{{ i }}:{{ hostvars[i].ansible_host }}"
{% endfor %}
    container_name: stolon-proxy
    restart: always
    command:
      - gosu
      - stolon
      - stolon-proxy
      - --listen-address=0.0.0.0
      - --cluster-name={{ stolon_cluster_name }}
      - --store-backend=etcdv3
      - --store-endpoints={% for i in groups['stolon'] %}http://{{ i }}:2379{{ "," if not loop.last else "\n" }}{% endfor %}
      - --log-level=info
    ports:
      - 5432:5432

volumes:
  pgkeeper:
    driver: local
#3

That’s nice! So you don’t need to install stolon as a linux service and use docker-compose, I like it.
I’ll try this out and also will look for articles about running stolon in docker-compose w/o swarm.
Do you know any already?