Last Class: Fault ToleranceToday: More on Fault ToleranceReliable One-One CommunicationReliable One-Many CommunicationAtomic MulticastImplementing Virtual Synchrony in IsisDistributed CommitTwo Phase CommitImplementing Two-Phase CommitImplementing 2PCThree-Phase CommitRecoveryIndependent CheckpointingCoordinated CheckpointingMessage LoggingCS677: Distributed OSComputer ScienceLecture 18, page 1Last Class: Fault Tolerance•Basic concepts and failure models•Failure masking using redundancy•Agreement in presence of faults–Two army problem–Byzantine generals problemCS677: Distributed OSComputer ScienceLecture 18, page 2Today: More on Fault Tolerance•Reliable communication–One-one communication–One-many communication•Distributed commit–Two phase commit–Three phase commit•Failure recovery–Checkpointing–Message loggingCS677: Distributed OSComputer ScienceLecture 18, page 3Reliable One-One Communication•Issues were discussed in Lecture 3–Use reliable transport protocols (TCP) or handle at the application layer•RPC semantics in the presence of failures•Possibilities–Client unable to locate server–Lost request messages–Server crashes after receiving request–Lost reply messages–Client crashes after sending requestCS677: Distributed OSComputer ScienceLecture 18, page 4Reliable One-Many Communication•Reliable multicast–Lost messages => need to retransmit•Possibilities–ACK-based schemes•Sender can become bottleneck–NACK-based schemesCS677: Distributed OSComputer ScienceLecture 18, page 5Atomic Multicast•Atomic multicast: a guarantee that all process received the message or none at all–Replicated database example•Problem: how to handle process crashes?•Solution: group view–Each message is uniquely associated with a group of processes•View of the process group when message was sent•All processes in the group should have the same view (and agree on it)Virtually Synchronous MulticastCS677: Distributed OSComputer ScienceLecture 18, page 6Implementing Virtual Synchrony in Isisa) Process 4 notices that process 7 has crashed, sends a view changeb) Process 6 sends out all its unstable messages, followed by a flush messagec) Process 6 installs the new view when it has received a flush message from everyone elseCS677: Distributed OSComputer ScienceLecture 18, page 7Distributed Commit•Atomic multicast example of a more general problem–All processes in a group perform an operation or not at all–Examples: •Reliable multicast: Operation = delivery of a message•Distributed transaction: Operation = commit transaction•Problem of distributed commit–All or nothing operations in a group of processes•Possible approaches–Two phase commit (2PC) [Gray 1978 ]–Three phase commitCS677: Distributed OSComputer ScienceLecture 18, page 8Two Phase Commit•Coordinator process coordinates the operation•Involves two phases–Voting phase: processes vote on whether to commit–Decision phase: actually commit or abortCS677: Distributed OSComputer ScienceLecture 18, page 9Implementing Two-Phase Commit•Outline of the steps taken by the coordinator in a two phase commit protocolactions by coordinator:while START _2PC to local log;multicast VOTE_REQUEST to all participants;while not all votes have been collected { wait for any incoming vote; if timeout { while GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants; exit; } record vote;}if all participants sent VOTE_COMMIT and coordinator votes COMMIT{ write GLOBAL_COMMIT to local log; multicast GLOBAL_COMMIT to all participants;} else { write GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants;}CS677: Distributed OSComputer ScienceLecture 18, page 10Implementing 2PCactions by participant:write INIT to local log;wait for VOTE_REQUEST from coordinator;if timeout { write VOTE_ABORT to local log; exit;}if participant votes COMMIT { write VOTE_COMMIT to local log; send VOTE_COMMIT to coordinator; wait for DECISION from coordinator; if timeout { multicast DECISION_REQUEST to other participants; wait until DECISION is received; /* remain blocked */ write DECISION to local log; } if DECISION == GLOBAL_COMMIT write GLOBAL_COMMIT to local log; else if DECISION == GLOBAL_ABORT write GLOBAL_ABORT to local log;} else { write VOTE_ABORT to local log; send VOTE ABORT to coordinator;}actions for handling decision requests: /*executed by separate thread */while true { wait until any incoming DECISION_REQUEST is received; /* remain blocked */ read most recently recorded STATE from the local log; if STATE == GLOBAL_COMMIT send GLOBAL_COMMIT to requesting participant; else if STATE == INIT or STATE == GLOBAL_ABORT send GLOBAL_ABORT to requesting participant; else skip; /* participant remains blocked */CS677: Distributed OSComputer ScienceLecture 18, page 11Three-Phase CommitTwo phase commit: problem if coordinator crashes (processes block)Three phase commit: variant of 2PC that avoids blockingCS677: Distributed OSComputer ScienceLecture 18, page 12Recovery•Techniques thus far allow failure handling•Recovery: operations that must be performed after a failure to recover to a correct state•Techniques:–Checkpointing:•Periodically checkpoint state •Upon a crash roll back to a previous checkpoint with a consistent stateCS677: Distributed OSComputer ScienceLecture 18, page 13Independent Checkpointing•Each processes periodically checkpoints independently of other processes•Upon a failure, work backwards to locate a consistent cut•Problem: if most recent checkpoints form inconsistenct cut, will need to keep rolling back until a consistent cut is found•Cascading rollbacks can lead to a domino effect.CS677: Distributed OSComputer ScienceLecture 18, page 14Coordinated Checkpointing•Take a distributed snapshot [discussed in Lec 11]•Upon a failure, roll back to the latest snapshot –All process restart from the latest snapshotCS677: Distributed OSComputer ScienceLecture 18, page 15Message Logging•Checkpointing is expensive–All processes restart from previous consistent cut–Taking a snapshot is expensive–Infrequent snapshots => all computations after previous snapshot will need to be redone [wasteful]•Combine checkpointing (expensive) with message logging (cheap)–Take infrequent checkpoints–Log all messages between checkpoints to local stable storage–To recover: simply replay messages