Failure detection and leases

Failure detection is how machines in a cluster decide that one of them has died, usually by expecting regular heartbeat messages and declaring a node dead after a timeout. Because that’s only a guess, two nodes can end up both believing they’re in charge, a situation called split brain. A lease limits the damage: it’s a lock with an expiry time, so a holder that crashes or gets cut off loses it automatically. But a holder can be frozen, say by a long garbage-collection pause, and wake up still thinking its lease is valid. The fix is a fencing token: a number that goes up with every new lease, which the holder sends with each write, so storage can reject any write carrying an older number than one it has already seen.