From time to time, one needs to set up new followers—perhaps to increase the number of replicas, or to replace failed nodes. How is it ensured that the new follower has an accurate copy of the leader?
Above statement is picked from DDIA(by Martin Klepmann).Let's dissect it
Take a Consistent Snapshot of the Leader’s Database:
Capture a consistent snapshot of the leader's database at a specific point in time.
If possible, avoid locking the entire database during this process to minimize disruption to active queries.
Many databases support this feature natively (as it is often used for backups). For example:
PostgreSQL provides tools for consistent snapshots using features like
pg_basebackup.MySQL may require third-party tools like
innobackupexfor consistent snapshot creation.
Copy the Snapshot to the New Follower Node:
Transfer the captured snapshot to the new follower node.
Ensure the snapshot contains the complete state of the database at the time it was captured.
Synchronize the Follower with the Leader’s Replication Log:
The follower connects to the leader and requests all the data changes that occurred since the snapshot was taken.
The snapshot must be associated with an exact position in the leader’s replication log to track subsequent changes.
Example terms for this position include:
PostgreSQL: Log Sequence Number (LSN).
MySQL: Binlog Coordinates.
Process the Backlog of Changes:
The follower processes the backlog of changes from the leader’s replication log.
These changes include all the updates, deletes, and inserts that occurred after the snapshot.
Catch Up:
Once the follower has processed all changes from the backlog and is synchronized with the leader, it is considered to have caught up.
From this point onward, the follower continuously replicates new changes from the leader in real-time.
This approach ensures that the follower node is brought online efficiently without disrupting the leader, maintaining data consistency throughout the process.
