The required feature described as a wish
Today if a KVM host fails and the management server and/or database is unreachable at the same moment (a remote site outage, a network partition, or the management server itself running as a guest on the failing host) CloudStack cannot restart the affected VMs. Every HA and DRS decision path runs inside the management server process and depends on the database.
I propose to add a small, storage-aware cluster mechanism that can restart a VM on a surviving host using its own quorum and fencing, and hands control back cleanly once the management server returns. This might require the Agents to actually keep shadow copy of each of the VMs configuration (of just enough per-VM data to rebuild the libvirt domain: disk identifiers, NIC MACs, CPU/memory, bridge/VLAN mapping). Possibly the agents can exchange information with each other and in case one Agent looses connectivity to management server act as a proxy towards the management server?
This might be an addition/similar or even fix to the #13589 , #14206 , or rely on some other mechanisms as surviving hosts must agree "host A (and possibly the management server) is gone" without a central arbiter and must not false-positive on an ordinary network hiccup,
Options I see as viable:
- Generic - Corosync for membership, Pacemaker for orchestration, IPMI/BMC STONITH for fencing.
- Storage specific - LINSTOR + drbd-reactor, CloudStack already ships a maintained LINSTOR primary storage plugin whose KVM-side driver talks straight to the LINSTOR controller, bypassing the management server for volume attach, and drbd quorum enforces split-brain protection, so it might be a nice addition.
The required feature described as a wish
Today if a KVM host fails and the management server and/or database is unreachable at the same moment (a remote site outage, a network partition, or the management server itself running as a guest on the failing host) CloudStack cannot restart the affected VMs. Every HA and DRS decision path runs inside the management server process and depends on the database.
I propose to add a small, storage-aware cluster mechanism that can restart a VM on a surviving host using its own quorum and fencing, and hands control back cleanly once the management server returns. This might require the Agents to actually keep shadow copy of each of the VMs configuration (of just enough per-VM data to rebuild the libvirt domain: disk identifiers, NIC MACs, CPU/memory, bridge/VLAN mapping). Possibly the agents can exchange information with each other and in case one Agent looses connectivity to management server act as a proxy towards the management server?
This might be an addition/similar or even fix to the #13589 , #14206 , or rely on some other mechanisms as surviving hosts must agree "host A (and possibly the management server) is gone" without a central arbiter and must not false-positive on an ordinary network hiccup,
Options I see as viable: