> Source: https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Troubleshooting/EtcdDebug/

# Troubleshooting `etcd`

> **Information**
>
> The whole procedure must be followed as `root`. To elevate your privileges to `root`, you can use the `su -` command.

`etcd` is a distributed key-value store used as the coordination point between the nodes of the PostgreSQL cluster. It reliably centralizes and shares the state of the cluster, including the identity of the leader, the state of the members and the locking mechanisms.

`patroni` relies on `etcd` to drive the election of the leader node and to trigger failovers automatically when an incident occurs. Installed on all nodes, `etcd` guarantees the overall consistency of the cluster, without taking part in storing or processing PostgreSQL data.

`patroni` therefore depends on `etcd`, and cannot work properly if `etcd` is not working.

> **Information**
>
> The `etcd` configuration is stored in the `/etc/default/etcd` file on each node.

## Logs

The whole procedure relies on the logs of the `etcd` service, which you get with the command below:

```
1
```

> **Tip**
>
> To follow the logs live, add the `-f` option to the command:
>
> ```
> 1
> ```

> **Information**
>
> The `etcd` logs are also available in `/var/log/syslog`.

## Error log explanations

> **`health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout`**
>
> If the `TCP/2380` flow is not open between two nodes, the connection times out and `etcd` writes the following logs:
>
> ```
> 1
> 2
> 3
> 4
> 5
> 6
> ```
>
> Where `<PEER_PSQL_NODE_ID>` is the ID assigned by `etcd` to the target node, and `<PEER_IP_PSQL_NODE>` the IP address of the target node.
>
> > **Example**
> >
> > In our example, the PostgreSQL cluster is configured as follows:
> >
> > | Node | Node IP | `etcd` ID |
> > |---|---|---|
> > | PSQL_1 | 192.168.1.1 | 53d2c129945ccb8b |
> > | PSQL_2 | 192.168.1.2 | 7176dd381f583d83 |
> > | PSQL_3 | 192.168.1.3 | e3d5ef565a5bb46c |
> >
> > Running the `journalctl -fu etcd` command from the `PSQL_1` server gives the following logs:
> >
> > ```
> > 1
> > 2
> > 3
> > 4
> > 5
> > ```
> >
> > These logs indicate that the connection from the `PSQL_1` server to the server with IP address 192.168.1.2 (`PSQL_2`) and `etcd` ID 7176dd381f583d83, on port `TCP/2380`, did not succeed: it timed out. We can deduce that the flow from the `PSQL_1` server to the `PSQL_2` server is dropped (`DROP`) by a firewall.
>
> This flow can be dropped (`DROP`) by the local firewall of the PostgreSQL appliance: check the [appliance firewall configuration](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Operation/PsqlFirewallUpdates/) on the nodes involved. If this configuration is correct, a firewall of the infrastructure is dropping (`DROP`) the `TCP/2380` flow between the two nodes.

> **`health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused`**
>
> When a connection between two nodes is refused, `etcd` reports it with the following logs:
>
> ```
> 1
> 2
> 3
> 4
> 5
> 6
> ```
>
> Where `<PEER_PSQL_NODE_ID>` is the ID assigned by `etcd` to the target node, and `<PEER_IP_PSQL_NODE>` the IP address of the target node.
>
> The refusal can have several causes:
>
> - The `etcd` service is malfunctioning or stopped on the target node. In this case, check the status (`systemctl status etcd`) and the logs of the `etcd` service on that node.
> - A firewall rejects (`REJECT`) the `TCP/2380` flow between the two nodes. It can be the [local firewall of the appliance](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Operation/PsqlFirewallUpdates/) or a firewall of the infrastructure.
>
> > **Example**
> >
> > In our example, the PostgreSQL cluster is configured as follows:
> >
> > | Node | Node IP | `etcd` ID |
> > |---|---|---|
> > | PSQL_1 | 192.168.1.1 | 53d2c129945ccb8b |
> > | PSQL_2 | 192.168.1.2 | 7176dd381f583d83 |
> > | PSQL_3 | 192.168.1.3 | e3d5ef565a5bb46c |
> >
> > Running the `journalctl -fu etcd` command from the `PSQL_1` server gives the following logs:
> >
> > ```
> > 1
> > 2
> > 3
> > 4
> > 5
> > ```
> >
> > These logs indicate that the connection from the `PSQL_1` server to the server with IP address 192.168.1.3 (`PSQL_3`) and `etcd` ID e3d5ef565a5bb46c, on port `TCP/2380`, was refused.
> >
> > On the `PSQL_3` server, the `systemctl status etcd` command shows that `etcd` is stopped:
> >
> > ```
> >  1
> >  2
> >  3
> >  4
> >  5
> >  6
> >  7
> >  8
> >  9
> > 10
> > 11
> > 12
> > 13
> > 14
> > 15
> > 16
> > 17
> > 18
> > 19
> > 20
> > ```
> >
> > Once `etcd` is started, the problem is solved.

> **`rejected connection from "<PEER_IP_PSQL_NODE>:35956" (error "remote error: tls: bad certificate", ServerName "PSQL_1")`**
>
> If the PostgreSQL cluster certificates have expired, or if they were not properly installed or renewed on the node from which you are reading the logs, the following error appears in the `etcd` logs:
>
> ```
> 1
> 2
> 3
> 4
> 5
> 6
> ```
>
> Where `<PEER_IP_PSQL_NODE>` is the IP address of one of the cluster nodes.
>
> This log indicates that, when the node you are connected to tried to establish a connection with the other cluster nodes (`<PEER_IP_PSQL_NODE>`), they refused it, reporting an error on the certificate of the current node.
>
> You should therefore [check the validity of the cluster certificates](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Operation/PsqlCertsVerification/) on the node you are connected to.
>
> > **Warning**
> >
> > If the certification authority configured on a remote node (in `/etc/etcd/ca.crt`) is incorrect, the error also appears on the node from which you are reading the logs: it then indicates that the remote node cannot validate the certificate presented, because it does not have the right CA.
>
> > **Example**
> >
> > In our example, the PostgreSQL cluster is configured as follows:
> >
> > | Node | Node IP | `etcd` ID |
> > |---|---|---|
> > | PSQL_1 | 192.168.1.1 | 53d2c129945ccb8b |
> > | PSQL_2 | 192.168.1.2 | 7176dd381f583d83 |
> > | PSQL_3 | 192.168.1.3 | e3d5ef565a5bb46c |
> >
> > Running the `journalctl -fu etcd` command from the `PSQL_1` server gives the following logs:
> >
> > ```
> > 1
> > 2
> > 3
> > 4
> > 5
> > 6
> > ```
> >
> > These logs indicate that the `PSQL_2` and `PSQL_3` nodes refused the connection from the `PSQL_1` node, because the certificate of `PSQL_1` is not valid in this context.
> >
> > We therefore check the validity of the `PSQL_1` node certificates by following the [cluster certificate check](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Operation/PsqlCertsVerification/) procedure:
> >
> > ```
> > 1
> > 2
> > ```
> >
> > The output shows that the node certificate has expired:
> >
> > ```
> > 1
> > 2
> > ```
> >
> > In this case, you need to [renew the PostgreSQL cluster certificates](https://docs.cyberelements.io/bastion/4.6/Installation/Dependencies/PostgreSQLCluster/Operation/PsqlCertsRenew/).

> **`error listing data dir: /var/lib/etcd/cleanroom`**
>
> If `etcd` does not have the right permissions on the `/var/lib/etcd/cleanroom` directory, the service refuses to start and writes the following logs:
>
> ```
> 1
> 2
> 3
> 4
> 5
> 6
> ```
>
> To solve this problem, assign the right permissions to the directory with the following command:
>
> ```
> 1
> ```
