Longhorn-aware drain deadlocks: node stuck cordoned, never reboots #1

New issue

Open

opened 2026-08-23 07:56:53 +00:00 by felix · 1 comment

felix commented

2026-08-23 07:56:53 +00:00

Owner

Image: krautit/fleetlock:0.4.0-longhorn-aware-drain2

Symptom: A node granted a reboot lease gets stuck cordoned/drained indefinitely, retrying every ~5 min without ever rebooting.

Root cause: The reboot flow checks Longhorn volume health after cordoning and draining the node:

Zincati requests a reboot lease → fleetlock grants it.
fleetlock cordons + drains the node.
fleetlock checks Longhorn volume robustness, refuses to proceed: node not ready to reboot, longhorn volume <pvc> is degraded.
But those volumes are degraded because the node was just cordoned in step 2 — Longhorn can't place their 3rd replica on a cordoned node.
Since the node stays cordoned while waiting for volumes to become healthy, and volumes can't become healthy while the node is cordoned, this never resolves on its own.

Observed stuck for 1h40m+, retrying every ~5 min, until manually uncordoned.

Evidence (repeats identically every ~5 min from 06:08 UTC 2026-08-23):

level=info msg="drainer: cordoning node" node=junta.kubo
level=info msg="drainer: draining node" node=junta.kubo
level=info msg="drainer: drained node" node=junta.kubo
level=info msg="fleetlock: node not ready to reboot, longhorn volume pvc-02ed96b5-... is degraded" ...

Suggested fix: Check Longhorn volume health before cordoning/draining, not after. If volumes hosted on this node are healthy pre-drain, the health check should either be skipped post-drain (since cordoning inherently removes a replica target) or should tolerate "degraded solely due to this node's own cordon" as an acceptable state, rather than blocking forever.

Workaround applied: manually ran kubectl uncordon <node> to let Longhorn rebuild the missing replica, which let the next fleetlock retry pass the health check.

**Image:** `krautit/fleetlock:0.4.0-longhorn-aware-drain2` **Symptom:** A node granted a reboot lease gets stuck cordoned/drained indefinitely, retrying every ~5 min without ever rebooting. **Root cause:** The reboot flow checks Longhorn volume health *after* cordoning and draining the node: 1. Zincati requests a reboot lease → fleetlock grants it. 2. fleetlock cordons + drains the node. 3. fleetlock checks Longhorn volume robustness, refuses to proceed: `node not ready to reboot, longhorn volume <pvc> is degraded`. 4. But those volumes are degraded **because** the node was just cordoned in step 2 — Longhorn can't place their 3rd replica on a cordoned node. 5. Since the node stays cordoned while waiting for volumes to become healthy, and volumes can't become healthy while the node is cordoned, this never resolves on its own. Observed stuck for 1h40m+, retrying every ~5 min, until manually uncordoned. **Evidence (repeats identically every ~5 min from 06:08 UTC 2026-08-23):** ``` level=info msg="drainer: cordoning node" node=junta.kubo level=info msg="drainer: draining node" node=junta.kubo level=info msg="drainer: drained node" node=junta.kubo level=info msg="fleetlock: node not ready to reboot, longhorn volume pvc-02ed96b5-... is degraded" ... ``` **Suggested fix:** Check Longhorn volume health *before* cordoning/draining, not after. If volumes hosted on this node are healthy pre-drain, the health check should either be skipped post-drain (since cordoning inherently removes a replica target) or should tolerate "degraded solely due to this node's own cordon" as an acceptable state, rather than blocking forever. **Workaround applied:** manually ran `kubectl uncordon <node>` to let Longhorn rebuild the missing replica, which let the next fleetlock retry pass the health check.

felix commented

2026-08-23 08:01:01 +00:00

Author

Owner

Update: pulled the full 24h log history from SigNoz. Confirms this consumed the entire night's reboot window on junta.kubo — no other node had a pending update to compare against.

Exact stuck error message (repeated verbatim every ~5 min from 02:16:37 UTC to 07:48:52 UTC, 65 times):

level=info msg="fleetlock: node not ready to reboot, longhorn volume pvc-02ed96b5-43c3-4177-8676-a758ab87665e is degraded" group=default holder=dd5a375e77aa438fb5824ace4976e673 id=dd5a375e77aa438fb5824ace4976e673

Timeline for junta.kubo:

19:37:58 (Aug 22) — Zincati requests reboot, fleetlock obtains the lease. Held outside the reboot window (74x "outside the reboot window" logged while waiting).
02:00:21 — window opens (04:00 Berlin), fleetlock cordons the node for the first time.
02:05:44–02:11:07 — drain blocked by PodDisruptionBudgets on 3 single-instance CNPG Postgres pods (pid-kellner-id-cnpg-2, pid-kraut-id-cnpg-3, immich-cnpg-1), plus a wider wait for 13 pods total initially.
02:16:37 — drain finally succeeds, but immediately hits the Longhorn-degraded check and blocks.
02:16:37 → 07:48:52 — stuck retrying that exact same check every ~5 min for 5.5 hours, never rebooting.
07:52:27 — fleetlock pod restarted (right after we manually ran kubectl uncordon junta.kubo), breaking the loop.

So the node never got its pending OS update applied last night — the drain succeeded, but the post-drain Longhorn health check deadlocked against itself for the rest of the window.

**Update:** pulled the full 24h log history from SigNoz. Confirms this consumed the entire night's reboot window on `junta.kubo` — no other node had a pending update to compare against. **Exact stuck error message** (repeated verbatim every ~5 min from 02:16:37 UTC to 07:48:52 UTC, 65 times): ``` level=info msg="fleetlock: node not ready to reboot, longhorn volume pvc-02ed96b5-43c3-4177-8676-a758ab87665e is degraded" group=default holder=dd5a375e77aa438fb5824ace4976e673 id=dd5a375e77aa438fb5824ace4976e673 ``` **Timeline for `junta.kubo`:** - **19:37:58** (Aug 22) — Zincati requests reboot, fleetlock obtains the lease. Held outside the reboot window (74x "outside the reboot window" logged while waiting). - **02:00:21** — window opens (04:00 Berlin), fleetlock cordons the node for the first time. - **02:05:44–02:11:07** — drain blocked by PodDisruptionBudgets on 3 single-instance CNPG Postgres pods (`pid-kellner-id-cnpg-2`, `pid-kraut-id-cnpg-3`, `immich-cnpg-1`), plus a wider wait for 13 pods total initially. - **02:16:37** — drain finally succeeds, but immediately hits the Longhorn-degraded check and blocks. - **02:16:37 → 07:48:52** — stuck retrying that exact same check every ~5 min for 5.5 hours, never rebooting. - **07:52:27** — fleetlock pod restarted (right after we manually ran `kubectl uncordon junta.kubo`), breaking the loop. So the node never got its pending OS update applied last night — the drain succeeded, but the post-drain Longhorn health check deadlocked against itself for the rest of the window.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

krautit/fleetlock#1

No description provided.

Rows
Columns