Preskoči na sadržaj

Otklanjanje poteškoća s uslugom etcd

Informacije

Cijeli postupak treba provesti kao root. Da biste povisili svoje povlastice na razinu root, možete upotrijebiti naredbu su -.

etcd je distribuirano spremište tipa ključ-vrijednost koje služi kao točka koordinacije između čvorova klastera PostgreSQL. Pouzdano centralizira i dijeli stanje klastera, uključujući identitet vodećeg čvora, stanje članova i mehanizme zaključavanja.

patroni se oslanja na etcd kako bi upravljao izborom vodećeg čvora i automatski pokretao prebacivanje u slučaju kvara kada dođe do incidenta. Instaliran na svim čvorovima, etcd jamči ukupnu dosljednost klastera, a da pritom ne sudjeluje u pohrani ni obradi podataka PostgreSQL.

patroni stoga ovisi o etcd i ne može ispravno raditi ako etcd ne radi.

Informacije

Konfiguracija usluge etcd pohranjena je u datoteci /etc/default/etcd na svakom čvoru.

Logs

Cijeli se postupak oslanja na zapisnike usluge etcd, koje dobivate naredbom u nastavku:

1
journalctl -u etcd

Savjet

Da biste pratili zapisnike uživo, dodajte naredbi mogućnost -f: 

1
journalctl -fu etcd

Informacije

Zapisnici usluge etcd dostupni su i u /var/log/syslog.

Objašnjenja zapisnika pogrešaka

health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout

Ako tok TCP/2380 nije otvoren između dvaju čvorova, vrijeme za povezivanje istječe, a etcd u zapisnik upisuje sljedeće:

1
2
3
4
5
6
Apr 15 11:48:09 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:09 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:14 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:14 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:19 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:19 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout

Pri čemu je <PEER_PSQL_NODE_ID> identifikator koji je etcd dodijelio ciljnom čvoru, a <PEER_IP_PSQL_NODE> IP adresa ciljnog čvora.

Primjer

U našem primjeru klaster PostgreSQL konfiguriran je ovako:

Čvor IP adresa čvora ID etcd
PSQL_1 192.168.1.1 53d2c129945ccb8b
PSQL_2 192.168.1.2 7176dd381f583d83
PSQL_3 192.168.1.3 e3d5ef565a5bb46c

Pokretanje naredbe journalctl -fu etcd na poslužitelju PSQL_1 daje sljedeće unose u zapisniku:

1
2
3
4
5
Apr 15 11:50:39 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout
Apr 15 11:50:44 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout
Apr 15 11:50:44 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout
Apr 15 11:50:49 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout
Apr 15 11:50:49 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout

Ti unosi pokazuju da povezivanje s poslužitelja PSQL_1 na poslužitelj s IP adresom 192.168.1.2 (PSQL_2) i identifikatorom etcd 7176dd381f583d83, na priključku TCP/2380, nije uspjelo: isteklo je vrijeme čekanja. Iz toga možemo zaključiti da vatrozid odbacuje (DROP) tok s poslužitelja PSQL_1 na poslužitelj PSQL_2.

Taj tok može odbacivati (DROP) lokalni vatrozid za appliance PostgreSQL: provjerite konfiguraciju vatrozida za appliance na dotičnim čvorovima. Ako je ta konfiguracija ispravna, tok TCP/2380 između dvaju čvorova odbacuje (DROP) vatrozid infrastrukture.

health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused

Kada se veza između dvaju čvorova odbije, etcd to prijavljuje sljedećim unosima u zapisniku:

1
2
3
4
5
6
Apr 15 14:04:30 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:30 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:35 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:35 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:40 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:40 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused

Pri čemu je <PEER_PSQL_NODE_ID> identifikator koji je etcd dodijelio ciljnom čvoru, a <PEER_IP_PSQL_NODE> IP adresa ciljnog čvora.

Odbijanje može imati nekoliko uzroka:

  • Usluga etcd ne radi ispravno ili je zaustavljena na ciljnom čvoru. U tom slučaju provjerite stanje (systemctl status etcd) i zapisnike usluge etcd na tom čvoru.
  • Vatrozid odbija (REJECT) tok TCP/2380 između dvaju čvorova. To može biti lokalni vatrozid za appliance ili vatrozid infrastrukture.
Primjer

U našem primjeru klaster PostgreSQL konfiguriran je ovako:

Čvor IP adresa čvora ID etcd
PSQL_1 192.168.1.1 53d2c129945ccb8b
PSQL_2 192.168.1.2 7176dd381f583d83
PSQL_3 192.168.1.3 e3d5ef565a5bb46c

Pokretanje naredbe journalctl -fu etcd na poslužitelju PSQL_1 daje sljedeće unose u zapisniku:

1
2
3
4
5
Apr 15 14:04:15 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused
Apr 15 14:04:15 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused
Apr 15 14:04:20 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused
Apr 15 14:04:20 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused
Apr 15 14:04:25 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused

Ti unosi pokazuju da je povezivanje s poslužitelja PSQL_1 na poslužitelj s IP adresom 192.168.1.3 (PSQL_3) i identifikatorom etcd e3d5ef565a5bb46c, na priključku TCP/2380, odbijeno.

Na poslužitelju PSQL_3 naredba systemctl status etcd pokazuje da je etcd zaustavljen:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
○ etcd.service - etcd - highly-available key value store
    Loaded: loaded (/lib/systemd/system/etcd.service; enabled; preset: enabled)
    Active: inactive (dead) since Wed 2026-04-15 16:09:34 CEST; 38s ago
Duration: 1min 36.087s
    Docs: https://etcd.io/docs
            man:etcd
    Process: 1400558 ExecStart=/usr/bin/etcd $DAEMON_ARGS (code=killed, signal=TERM)
Main PID: 1400558 (code=killed, signal=TERM)
        CPU: 49.071s

Apr 15 12:26:32 PSQL_3 etcd[1197556]: finished scheduled compaction at 1665243 (took 583.297µs)
Apr 15 13:26:32 PSQL_3 etcd[1197556]: store.index: compact 1665949
Apr 15 13:26:32 PSQL_3 etcd[1197556]: finished scheduled compaction at 1665949 (took 574.496µs)
Apr 15 14:26:32 PSQL_3 etcd[1197556]: store.index: compact 1666658
Apr 15 14:26:32 PSQL_3 etcd[1197556]: finished scheduled compaction at 1666658 (took 567.196µs)
Apr 15 15:26:32 PSQL_3 etcd[1197556]: store.index: compact 1667368
Apr 15 15:26:32 PSQL_3 etcd[1197556]: finished scheduled compaction at 1667368 (took 568.096µs)
Apr 15 16:09:34 PSQL_3 systemd[1]: etcd.service: Deactivated successfully.
Apr 15 16:09:34 PSQL_3 systemd[1]: Stopped etcd.service - etcd - highly-available key value store.
Apr 15 16:09:34 PSQL_3 systemd[1]: etcd.service: Consumed 49.071s CPU time.

Kada se etcd pokrene, problem je riješen.

rejected connection from "<PEER_IP_PSQL_NODE>:35956" (error "remote error: tls: bad certificate", ServerName "PSQL_1")

Ako su certifikati klastera PostgreSQL istekli ili nisu ispravno instalirani ili obnovljeni na čvoru s kojeg čitate zapisnike, u zapisnicima usluge etcd pojavljuje se sljedeća pogreška:

1
2
3
4
5
6
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49274" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49294" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49288" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49316" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49304" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:60958" (error "remote error: tls: bad certificate", ServerName "PSQL_1")

Pri čemu je <PEER_IP_PSQL_NODE> IP adresa jednog od čvorova klastera.

Ovaj unos pokazuje da su, kada je čvor s kojim ste povezani pokušao uspostaviti vezu s ostalim čvorovima klastera (<PEER_IP_PSQL_NODE>), ti čvorovi odbili vezu i prijavili pogrešku u certifikatu trenutačnog čvora.

Stoga provjerite valjanost certifikata klastera na čvoru s kojim ste povezani.

Upozorenje

Ako je certifikacijsko tijelo konfigurirano na udaljenom čvoru (u /etc/etcd/ca.crt) neispravno, pogreška se pojavljuje i na čvoru s kojeg čitate zapisnike: tada znači da udaljeni čvor ne može provjeriti predočeni certifikat jer nema ispravno certifikacijsko tijelo (CA).

Primjer

U našem primjeru klaster PostgreSQL konfiguriran je ovako:

Čvor IP adresa čvora ID etcd
PSQL_1 192.168.1.1 53d2c129945ccb8b
PSQL_2 192.168.1.2 7176dd381f583d83
PSQL_3 192.168.1.3 e3d5ef565a5bb46c

Pokretanje naredbe journalctl -fu etcd na poslužitelju PSQL_1 daje sljedeće unose u zapisniku:

1
2
3
4
5
6
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.2:49274" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.3:49294" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.2:49288" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.2:49316" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.3:49304" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.3:60958" (error "remote error: tls: bad certificate", ServerName "PSQL_1")

Ti unosi pokazuju da su čvorovi PSQL_2 i PSQL_3 odbili vezu s čvora PSQL_1 jer certifikat čvora PSQL_1 u tom kontekstu nije valjan.

Stoga provjeravamo valjanost certifikata čvora PSQL_1 slijedeći postupak provjere certifikata klastera:

1
2
openssl x509 -in /etc/etcd/peer.crt -noout -enddate
openssl x509 -in /etc/etcd/ca.crt -noout -enddate

Rezultat pokazuje da je certifikat čvora istekao:

1
2
notAfter=Dec 18 14:04:09 2025 GMT
notAfter=Dec 17 14:03:47 2030 GMT

U tom slučaju morate obnoviti certifikate klastera PostgreSQL.

error listing data dir: /var/lib/etcd/cleanroom

Ako etcd nema ispravne dozvole za direktorij /var/lib/etcd/cleanroom, usluga se odbija pokrenuti i u zapisnik upisuje sljedeće:

1
2
3
4
5
6
Apr 10 14:15:26 PSQL_1 etcd[454701]: Go OS/Arch: linux/amd64
Apr 10 14:15:26 PSQL_1 etcd[454701]: setting maximum number of CPUs to 1, total number of available CPUs is 1
Apr 10 14:15:26 PSQL_1 etcd[454701]: error listing data dir: /var/lib/etcd/cleanroom
Apr 10 14:15:26 PSQL_1 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Apr 10 14:15:26 PSQL_1 systemd[1]: etcd.service: Failed with result 'exit-code'.
Apr 10 14:15:26 PSQL_1 systemd[1]: Failed to start etcd.service - etcd - highly-available key value store

Da biste riješili taj problem, dodijelite direktoriju ispravne dozvole sljedećom naredbom:

1
chown -R etcd:etcd /var/lib/etcd/cleanroom