Troubleshooting patroni¶
Information
The whole procedure must be followed as root. To elevate your privileges to root, you can use the su - command.
Warning
To follow this procedure, you must have carried out the additional configurations for patronictl.
patroni is a high-availability orchestrator for PostgreSQL. It automates the management of a PostgreSQL cluster by handling leader election, failover and node synchronization when an incident occurs.
In the CyberElements Bastion PostgreSQL cluster, patroni relies on etcd to store and share the state of the cluster. It continuously monitors the state of the PostgreSQL instances and makes sure that only one node is designated leader at any given time.
If the leader node fails, patroni automatically orchestrates an election through the etcd distributed configuration store, in order to promote a new leader and keep the service available.
patroni therefore depends on etcd, and cannot work properly if etcd is not working.
Node desynchronization¶
A node of the PostgreSQL cluster can sometimes get out of sync. This is visible in the output of the patronictl -c /etc/patroni/config.yml topology command:
1 2 3 4 5 6 7 | |
In the TL column, node PSQL_1 is no longer on the same TL as the others. For the cluster to work, all nodes must be on the same TL: a difference indicates that one of the nodes is out of sync.
In this case, run the following command on the out-of-sync node:
1 | |
Where <MEMBER_PATRONI> is the name of the out-of-sync node, as shown in the Member column of the topology.
If several nodes are out of sync, resynchronize all those that are not on the highest TL.
Warning
If the out-of-sync node is the Leader node, data may be lost. This should never happen: the Leader node is expected to be on the highest TL.
Example
In the output of the patronictl -c /etc/patroni/config.yml topology command above, node PSQL_1 is out of sync. To resynchronize it, use the following command:
1 | |
Logs¶
The whole procedure relies on the logs of the patroni service, which you get with the command below:
1 | |
Tip
To follow the logs live, add the -f option to the command:
1 | |
Information
The patroni logs are also available in /var/log/syslog.
Error log explanations¶
ERROR: Request to server https://<PSQL_NODE_NAME>:2379 failed: MaxRetryError("HTTPSConnectionPool(host='<PSQL_NODE_NAME>', port=2379)
When etcd is not working or cannot be reached, patroni can write the following errors:
1 2 3 4 5 6 7 8 | |
<PSQL_NODE_NAME>-
Name of one of the nodes of the PostgreSQL cluster.
In this case, you need to troubleshoot etcd.