|
upstream contribution · merged ✓
one bad node killed every egressip in the cluster
github.com/ovn-kubernetes/ovn-kubernetes · PR #6773
|
|
🐣 first — what is "EgressIP"?
Pods normally share a node's IP for outbound traffic.
EgressIP gives pods a stable, dedicated source IP instead.
Firewalls and partners allowlist this IP → pods can reach external services.
The cluster manager assigns EgressIPs to eligible nodes using
each node's k8s.ovn.org/host-cidrs annotation.
|
|
|
🐛 two problems found
|
1. 🔴 conflict check aborts on one bad node
isEgressIPAddrConflict() reads every node's
host-cidrs annotation.
If any node's annotation is missing or malformed → returns error.
assignEgressIPs() sees the error → aborts the entire pass.
every EgressIP in the cluster goes unassigned → cluster-wide egress outage
|
|
2. bad nodes stay in the assignment pool
A node with the egress-assignable label but
broken host-cidrs
was still considered a valid target for new EgressIP assignments.
IPs assigned to unvalidatable nodes → inconsistent state.
existing assignments on the bad node were never released
|
|
|
|
💭 what changed?
|
before fix
node-3 loses host-cidrs
↓
conflict check → error
↓
assignEgressIPs aborts
↓
ALL egressips: status = []
↓
entire cluster: no egress 💀
|
|
after fix
node-3 loses host-cidrs
↓
conflict check → skip + warn
↓
node-3 removed from pool
↓
existing IPs released → move
↓
other nodes: assigned ✓
|
|
|
|
💡 the rule
contain the blast radius.
When one component is broken, punish that component — not everyone.
A bad node is a bad node. It should not take the whole fleet down.
Skip, isolate, log, and let the healthy ones keep working.
Fail closed for the individual. Fail open for the system.
|
|
|
merged: Aug 24 2026 |
repo: ovn-kubernetes/ovn-kubernetes
|
pr: #6773
|
| small fix · big lesson |
contributed by Parikshit Khedekar |
|
|