Skip to content

Troubleshooting patroni

Information

The whole procedure must be followed as root. To elevate your privileges to root, you can use the su - command.

Warning

To follow this procedure, you must have carried out the additional configurations for patronictl.

patroni is a high-availability orchestrator for PostgreSQL. It automates the management of a PostgreSQL cluster by handling leader election, failover and node synchronization when an incident occurs.

In the CyberElements Bastion PostgreSQL cluster, patroni relies on etcd to store and share the state of the cluster. It continuously monitors the state of the PostgreSQL instances and makes sure that only one node is designated leader at any given time.

If the leader node fails, patroni automatically orchestrates an election through the etcd distributed configuration store, in order to promote a new leader and keep the service available.

patroni therefore depends on etcd, and cannot work properly if etcd is not working.

Node desynchronization

A node of the PostgreSQL cluster can sometimes get out of sync. This is visible in the output of the patronictl -c /etc/patroni/config.yml topology command:

1
2
3
4
5
6
7
+ Cluster: 15-cleanroomvault5 ------+---------+---------+----+-----------+
| Member           | Host           | Role    | State   | TL | Lag in MB |
+------------------+----------------+---------+---------+----+-----------+
| PSQL_3           | psql_3         | Leader  | running | 3  |           |
| + PSQL_1         | psql_1         | Replica | running | 2  |       230 |
| + PSQL_2         | psql_2         | Replica | running | 3  |         0 |
+------------------+----------------+---------+---------+----+-----------+

In the TL column, node PSQL_1 is no longer on the same TL as the others. For the cluster to work, all nodes must be on the same TL: a difference indicates that one of the nodes is out of sync.

In this case, run the following command on the out-of-sync node:

1
patronictl -c /etc/patroni/config.yml reinit 15-cleanroomvault5 <MEMBER_PATRONI>

Where <MEMBER_PATRONI> is the name of the out-of-sync node, as shown in the Member column of the topology.

If several nodes are out of sync, resynchronize all those that are not on the highest TL.

Warning

If the out-of-sync node is the Leader node, data may be lost. This should never happen: the Leader node is expected to be on the highest TL.

Example

In the output of the patronictl -c /etc/patroni/config.yml topology command above, node PSQL_1 is out of sync. To resynchronize it, use the following command:

1
patronictl -c /etc/patroni/config.yml reinit 15-cleanroomvault5 PSQL_1

Logs

The whole procedure relies on the logs of the patroni service, which you get with the command below:

1
journalctl -u patroni

Tip

To follow the logs live, add the -f option to the command:

1
journalctl -fu patroni

Information

The patroni logs are also available in /var/log/syslog.

Error log explanations

ERROR: Request to server https://<PSQL_NODE_NAME>:2379 failed: MaxRetryError("HTTPSConnectionPool(host='<PSQL_NODE_NAME>', port=2379)

When etcd is not working or cannot be reached, patroni can write the following errors:

1
2
3
4
5
6
7
8
patroni[1812916]: 2026-04-17 15:23:51,431 INFO: Retrying on https://<PSQL_NODE_NAME>:2379
patroni[1812916]: 2026-04-17 15:23:51,434 ERROR: Request to server https://<PSQL_NODE_NAME>:2379 failed: MaxRetryError("HTTPSConnectionPool(host='<PSQL_NODE_NAME>', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8924a49fd0>: Failed to establish a new connection: [Errno 111] Connection refused'))")
patroni[1812916]: 2026-04-17 15:23:51,434 INFO: Reconnection allowed, looking for another server.
patroni[1812916]: 2026-04-17 15:23:51,434 INFO: Retrying on https://<PSQL_NODE_NAME>:2379
patroni[1812916]: 2026-04-17 15:23:51,438 ERROR: Request to server https://<PSQL_NODE_NAME>:2379 failed: MaxRetryError("HTTPSConnectionPool(host='<PSQL_NODE_NAME>', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8924a48c50>: Failed to establish a new connection: [Errno 111] Connection refused'))")
patroni[1812916]: 2026-04-17 15:23:51,438 INFO: Reconnection allowed, looking for another server.
patroni[1812916]: 2026-04-17 15:23:53,940 ERROR: Request to server https://<PSQL_NODE_NAME>:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='<PSQL_NODE_NAME>', port=2379): Read timed out. (read timeout=2.0772262712319693)")
patroni[1812916]: 2026-04-17 15:23:53,940 INFO: Reconnection allowed, looking for another server.
<PSQL_NODE_NAME>

Name of one of the nodes of the PostgreSQL cluster.

In this case, you need to troubleshoot etcd.