> Source: https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Troubleshooting/PsqlDebug/

# Troubleshooting the PostgreSQL cluster

> **Information**
>
> The whole procedure must be followed as `root`. To elevate your privileges to `root`, you can use the `su -` command.

> **Warning**
>
> To follow this procedure, you must have carried out the [additional configurations for `patronictl`](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Installation/Install/#additional-configurations-for-patronictl).

Connection errors to the database of the CyberElements Bastion PostgreSQL cluster can have various causes. This section presents a general troubleshooting method, to be adapted to your case.

## Checking the cluster status

> **Information**
>
> Checking the cluster status quickly locates the failing node or nodes.

You can check the cluster status with the following command, to be run on each node of the PostgreSQL cluster:

```
1
```

> **Example of output on a working platform**
>
> ```
> 1
> 2
> 3
> 4
> 5
> 6
> 7
> ```
>
> In the output above, everything indicates that the PostgreSQL cluster is working: the lag is 0 MB for all `Replica` nodes, and all nodes are in the list, with the `running` status. You need to get this output on all nodes to confirm that `patroni` and `etcd` are working.

> **Example of output when there is a communication problem between the nodes**
>
> ```
> 1
> 2
> 3
> 4
> 5
> 6
> 7
> ```
>
> In the output above, the 230 MB lag is abnormal: it could indicate a communication problem between node 3 and node 1.
> If the problem persists after several hours, this confirms it.

> **Example of output when a node is not properly started or stopped**
>
> ```
> 1
> 2
> 3
> 4
> 5
> 6
> 7
> ```
>
> In the output above, the state (`State` column) indicates that node 1 is starting. If the `started` status persists for several minutes, or even hours, there may be a problem with `patroni` or `etcd` on that node.
> Node 2 is in the `stopped` state: it is stopped. When a cluster node fails, this information is only displayed temporarily, before the node is removed from the list.

> **Example of output when `etcd` fails on the current node**
>
> ```
> 1
> 2
> 3
> ```
>
> The output above indicates a connection error to `etcd`, which prevents retrieving the status of the PostgreSQL cluster.

> **Example of output when one of the nodes is out of sync**
>
> ```
> 1
> 2
> 3
> 4
> 5
> 6
> 7
> ```
>
> In the output above, node `PSQL_1` is no longer on the same `TL` as the others. For the cluster to work, all nodes must be on the same `TL`: a difference indicates that one of the nodes is out of sync.
> To solve this problem, follow the [node desynchronization](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Troubleshooting/PatroniDebug/#node-desynchronization) procedure.

## Checking the status of the services

Once the failing node has been identified, you need to identify the malfunctioning service. The two main services used on the PostgreSQL cluster are `patroni` and `etcd`.

> **Information**
>
> The `patroni` service depends on the `etcd` service: if `etcd` is not working, `patroni` will not work either.

To identify the failing service, you can run the following commands:

```
1
2
```

These commands give the status of the service, along with its last 10 log lines.

> **Warning**
>
> A service with the `active (running)` status is not necessarily working: it can be running while writing error logs. It is therefore important to read the service logs to understand its actual state.

> **Example of output for `etcd` when the service is working**
>
> ```
>  1
>  2
>  3
>  4
>  5
>  6
>  7
>  8
>  9
> 10
> 11
> 12
> 13
> 14
> 15
> 16
> 17
> 18
> 19
> 20
> 21
> 22
> ```

> **Example of output for `etcd` when the service is in error**
>
> ```
>  1
>  2
>  3
>  4
>  5
>  6
>  7
>  8
>  9
> 10
> 11
> 12
> 13
> 14
> 15
> 16
> 17
> 18
> 19
> 20
> ```
>
> In the output above, the `etcd` service has the `failed` status, and the error that prevented it from starting is the one on line 17: the service cannot read the content of the `/var/lib/etcd/cleanroom` directory. In this case, [adjust the permissions on the directory](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Troubleshooting/EtcdDebug/#error-log-explanations).

> **Example of output for `patroni` when the service is working**
>
> ```
>  1
>  2
>  3
>  4
>  5
>  6
>  7
>  8
>  9
> 10
> 11
> 12
> 13
> 14
> 15
> 16
> 17
> 18
> 19
> 20
> 21
> 22
> 23
> 24
> 25
> 26
> 27
> 28
> 29
> ```

> **Example of output for `patroni` when the service is in error**
>
> ```
>  1
>  2
>  3
>  4
>  5
>  6
>  7
>  8
>  9
> 10
> 11
> 12
> 13
> 14
> 15
> 16
> 17
> 18
> 19
> 20
> 21
> 22
> 23
> 24
> 25
> 26
> 27
> 28
> 29
> ```
>
> In the case above, the service has the `active (running)` status without actually working: the logs show various connection errors to `etcd`, which `patroni` needs.
> In this example, `patroni` is not necessarily at fault: since it depends on `etcd`, first fix the `etcd` problems, then check the status of `patroni` again.

## Service troubleshooting procedures

Below is one troubleshooting procedure per service. If both `etcd` and `patroni` are malfunctioning, start with `etcd`.

- **[Troubleshooting `etcd`](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Troubleshooting/EtcdDebug/)**

  ---

  Procedure to follow when the `etcd` service malfunctions.
- **[Troubleshooting `patroni`](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Troubleshooting/PatroniDebug/)**

  ---

  Procedure to follow when the `patroni` service malfunctions.
