Soluzzjoni tal-problemi ta’ etcd

Informazzjoni

Il-proċedura kollha għandha titwettaq bħala root. Biex tgħolli l-privileġġi tiegħek għal root, tista’ tuża l-kmand su -.

etcd huwa maħżen distribwit ta’ ċwievet u valuri li jintuża bħala l-punt ta’ koordinazzjoni bejn in-nodi tal-cluster PostgreSQL. B’mod affidabbli, jiċċentralizza u jaqsam l-istat tal-cluster, inklużi l-identità tan-nodu leader, l-istat tal-membri u l-mekkaniżmi ta’ lokkjar.

patroni jiddependi fuq etcd biex imexxi l-elezzjoni tan-nodu leader u biex jiskatta awtomatikament il-failover meta jseħħ inċident. Installat fuq in-nodi kollha, etcd jiggarantixxi l-konsistenza ġenerali tal-cluster, mingħajr ma jieħu sehem fil-ħażna jew fl-ipproċessar tad-data PostgreSQL.

Għalhekk patroni jiddependi fuq etcd, u ma jistax jaħdem sew jekk etcd ma jkunx qed jaħdem.

Informazzjoni

Il-konfigurazzjoni ta’ etcd hija maħżuna fil-fajl /etc/default/etcd fuq kull nodu.

Logs

Il-proċedura kollha hija bbażata fuq ir-reġistri tas-servizz etcd, li tiksbu bil-kmand ta’ hawn taħt:

journalctl -u etcd

Parir

Biex issegwi r-reġistri f’ħin reali, żid l-għażla -f mal-kmand:

journalctl -fu etcd

Informazzjoni

Ir-reġistri ta’ etcd huma disponibbli wkoll fil-fajl /var/log/syslog.

Spjegazzjoni tar-reġistri tal-iżbalji

health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout

Jekk il-fluss TCP/2380 ma jkunx miftuħ bejn żewġ nodi, il-konnessjoni taqbeż il-ħin massimu (timeout) u etcd jikteb ir-reġistri li ġejjin:

Apr 15 11:48:09 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:09 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:14 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:14 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:19 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout
Apr 15 11:48:19 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: i/o timeout

Fejn <PEER_PSQL_NODE_ID> huwa l-ID assenjat minn etcd lin-nodu tad-destinazzjoni, u <PEER_IP_PSQL_NODE> huwa l-indirizz IP tan-nodu tad-destinazzjoni.

Eżempju

Fl-eżempju tagħna, il-cluster PostgreSQL huwa kkonfigurat kif ġej:

Nodu	IP tan-nodu	ID ta’ etcd
PSQL_1	192.168.1.1	53d2c129945ccb8b
PSQL_2	192.168.1.2	7176dd381f583d83
PSQL_3	192.168.1.3	e3d5ef565a5bb46c

It-tħaddim tal-kmand journalctl -fu etcd mis-server PSQL_1 jagħti r-reġistri li ġejjin:

Apr 15 11:50:39 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout
Apr 15 11:50:44 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout
Apr 15 11:50:44 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout
Apr 15 11:50:49 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout
Apr 15 11:50:49 PSQL_1 etcd[1342933]: health check for peer 7176dd381f583d83 could not connect: dial tcp 192.168.1.2:2380: i/o timeout

Dawn ir-reġistri jindikaw li l-konnessjoni mis-server PSQL_1 għas-server bl-indirizz IP 192.168.1.2 (PSQL_2) u l-ID ta’ etcd 7176dd381f583d83, fuq il-port TCP/2380, ma rnexxietx: qabżet il-ħin massimu (timeout). Nistgħu nikkonkludu li l-fluss mis-server PSQL_1 għas-server PSQL_2 qed jiġi skartat (DROP) minn firewall.

Dan il-fluss jista’ jiġi skartat (DROP) mill-firewall lokali tal-apparat PostgreSQL: iċċekkja l-konfigurazzjoni tal-firewall tal-apparat fuq in-nodi kkonċernati. Jekk din il-konfigurazzjoni hija korretta, firewall tal-infrastruttura qed jiskarta (DROP) il-fluss TCP/2380 bejn iż-żewġ nodi.

health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused

Meta konnessjoni bejn żewġ nodi tiġi rrifjutata, etcd jirrapporta dan bir-reġistri li ġejjin:

Apr 15 14:04:30 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:30 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:35 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:35 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:40 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused
Apr 15 14:04:40 PSQL_1 etcd[1342933]: health check for peer <PEER_PSQL_NODE_ID> could not connect: dial tcp <PEER_IP_PSQL_NODE>:2380: connect: connection refused

Fejn <PEER_PSQL_NODE_ID> huwa l-ID assenjat minn etcd lin-nodu tad-destinazzjoni, u <PEER_IP_PSQL_NODE> huwa l-indirizz IP tan-nodu tad-destinazzjoni.

Ir-rifjut jista’ jkollu diversi kawżi:

• Is-servizz etcd mhux qed jaħdem sew jew huwa mwaqqaf fuq in-nodu tad-destinazzjoni. F’dan il-każ, iċċekkja l-istatus (systemctl status etcd) u r-reġistri tas-servizz etcd fuq dak in-nodu.
• Firewall jirrifjuta (REJECT) il-fluss TCP/2380 bejn iż-żewġ nodi. Jista’ jkun il-firewall lokali tal-apparat jew firewall tal-infrastruttura.

Eżempju

Fl-eżempju tagħna, il-cluster PostgreSQL huwa kkonfigurat kif ġej:

Nodu	IP tan-nodu	ID ta’ etcd
PSQL_1	192.168.1.1	53d2c129945ccb8b
PSQL_2	192.168.1.2	7176dd381f583d83
PSQL_3	192.168.1.3	e3d5ef565a5bb46c

It-tħaddim tal-kmand journalctl -fu etcd mis-server PSQL_1 jagħti r-reġistri li ġejjin:

Apr 15 14:04:15 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused
Apr 15 14:04:15 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused
Apr 15 14:04:20 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused
Apr 15 14:04:20 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused
Apr 15 14:04:25 PSQL_1 etcd[1342933]: health check for peer e3d5ef565a5bb46c could not connect: dial tcp 192.168.1.3:2380: connect: connection refused

Dawn ir-reġistri jindikaw li l-konnessjoni mis-server PSQL_1 għas-server bl-indirizz IP 192.168.1.3 (PSQL_3) u l-ID ta’ etcd e3d5ef565a5bb46c, fuq il-port TCP/2380, ġiet irrifjutata.

Fuq is-server PSQL_3, il-kmand systemctl status etcd juri li etcd huwa mwaqqaf:

○ etcd.service - etcd - highly-available key value store
    Loaded: loaded (/lib/systemd/system/etcd.service; enabled; preset: enabled)
    Active: inactive (dead) since Wed 2026-04-15 16:09:34 CEST; 38s ago
Duration: 1min 36.087s
    Docs: https://etcd.io/docs
            man:etcd
    Process: 1400558 ExecStart=/usr/bin/etcd $DAEMON_ARGS (code=killed, signal=TERM)
Main PID: 1400558 (code=killed, signal=TERM)
        CPU: 49.071s

Apr 15 12:26:32 PSQL_3 etcd[1197556]: finished scheduled compaction at 1665243 (took 583.297µs)
Apr 15 13:26:32 PSQL_3 etcd[1197556]: store.index: compact 1665949
Apr 15 13:26:32 PSQL_3 etcd[1197556]: finished scheduled compaction at 1665949 (took 574.496µs)
Apr 15 14:26:32 PSQL_3 etcd[1197556]: store.index: compact 1666658
Apr 15 14:26:32 PSQL_3 etcd[1197556]: finished scheduled compaction at 1666658 (took 567.196µs)
Apr 15 15:26:32 PSQL_3 etcd[1197556]: store.index: compact 1667368
Apr 15 15:26:32 PSQL_3 etcd[1197556]: finished scheduled compaction at 1667368 (took 568.096µs)
Apr 15 16:09:34 PSQL_3 systemd[1]: etcd.service: Deactivated successfully.
Apr 15 16:09:34 PSQL_3 systemd[1]: Stopped etcd.service - etcd - highly-available key value store.
Apr 15 16:09:34 PSQL_3 systemd[1]: etcd.service: Consumed 49.071s CPU time.

Ladarba etcd jinbeda, il-problema tissolva.

rejected connection from "<PEER_IP_PSQL_NODE>:35956" (error "remote error: tls: bad certificate", ServerName "PSQL_1")

Jekk iċ-ċertifikati tal-cluster PostgreSQL ikunu skadew, jew jekk ma ġewx installati jew imġedda sew fuq in-nodu minn fejn qed taqra r-reġistri, l-iżball li ġej jidher fir-reġistri ta’ etcd:

Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49274" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49294" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49288" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49316" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:49304" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "<PEER_IP_PSQL_NODE>:60958" (error "remote error: tls: bad certificate", ServerName "PSQL_1")

Fejn <PEER_IP_PSQL_NODE> huwa l-indirizz IP ta’ wieħed min-nodi tal-cluster.

Dan ir-reġistru jindika li, meta n-nodu li int ikkonnettjat miegħu pprova jistabbilixxi konnessjoni man-nodi l-oħra tal-cluster (<PEER_IP_PSQL_NODE>), dawn irrifjutawha, u rrappurtaw żball fuq iċ-ċertifikat tan-nodu attwali.

Għalhekk għandek tiċċekkja l-validità taċ-ċertifikati tal-cluster fuq in-nodu li int ikkonnettjat miegħu.

Twissija

Jekk l-awtorità taċ-ċertifikazzjoni kkonfigurata fuq nodu remot (fil-fajl /etc/etcd/ca.crt) mhix korretta, l-iżball jidher ukoll fuq in-nodu minn fejn qed taqra r-reġistri: f’dan il-każ jindika li n-nodu remot ma jistax jivvalida ċ-ċertifikat ippreżentat, għax m’għandux l-awtorità taċ-ċertifikazzjoni (CA) it-tajba.

Eżempju

Fl-eżempju tagħna, il-cluster PostgreSQL huwa kkonfigurat kif ġej:

Nodu	IP tan-nodu	ID ta’ etcd
PSQL_1	192.168.1.1	53d2c129945ccb8b
PSQL_2	192.168.1.2	7176dd381f583d83
PSQL_3	192.168.1.3	e3d5ef565a5bb46c

It-tħaddim tal-kmand journalctl -fu etcd mis-server PSQL_1 jagħti r-reġistri li ġejjin:

Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.2:49274" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.3:49294" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:54 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.2:49288" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.2:49316" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.3:49304" (error "remote error: tls: bad certificate", ServerName "PSQL_1")
Apr 15 16:38:55 PSQL_1 etcd[1407644]: rejected connection from "192.168.1.3:60958" (error "remote error: tls: bad certificate", ServerName "PSQL_1")

Dawn ir-reġistri jindikaw li n-nodi PSQL_2 u PSQL_3 irrifjutaw il-konnessjoni min-nodu PSQL_1, għax iċ-ċertifikat ta’ PSQL_1 mhuwiex validu f’dan il-kuntest.

Għalhekk niċċekkjaw il-validità taċ-ċertifikati tan-nodu PSQL_1 billi nsegwu l-proċedura ta’ verifika taċ-ċertifikati tal-cluster:

openssl x509 -in /etc/etcd/peer.crt -noout -enddate
openssl x509 -in /etc/etcd/ca.crt -noout -enddate

Ir-riżultat juri li ċ-ċertifikat tan-nodu skada:

notAfter=Dec 18 14:04:09 2025 GMT
notAfter=Dec 17 14:03:47 2030 GMT

F’dan il-każ, għandek iġġedded iċ-ċertifikati tal-cluster PostgreSQL.

error listing data dir: /var/lib/etcd/cleanroom

Jekk etcd m’għandux il-permessi t-tajbin fuq id-direttorju /var/lib/etcd/cleanroom, is-servizz jirrifjuta li jibda u jikteb ir-reġistri li ġejjin:

Apr 10 14:15:26 PSQL_1 etcd[454701]: Go OS/Arch: linux/amd64
Apr 10 14:15:26 PSQL_1 etcd[454701]: setting maximum number of CPUs to 1, total number of available CPUs is 1
Apr 10 14:15:26 PSQL_1 etcd[454701]: error listing data dir: /var/lib/etcd/cleanroom
Apr 10 14:15:26 PSQL_1 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Apr 10 14:15:26 PSQL_1 systemd[1]: etcd.service: Failed with result 'exit-code'.
Apr 10 14:15:26 PSQL_1 systemd[1]: Failed to start etcd.service - etcd - highly-available key value store

Biex issolvi din il-problema, assenja l-permessi t-tajbin lid-direttorju bil-kmand li ġej:

chown -R etcd:etcd /var/lib/etcd/cleanroom