What is ZK (Zookeeper) and how it is useful in design

ZooKeeper is a distributed coordination service that helps manage configuration, naming, synchronization, and group services in distributed systems.

1. Cluster Setup: ZooKeeper operates in a cluster of servers, with one server acting as the leader and others as followers.

   • Quorum: At least a majority of servers must be operational to make ZooKeeper available.

2. Client Interaction:

   • Clients connect to a ZooKeeper server in the cluster.

   • The server handles read requests directly.

   • For write requests, the server forwards them to the leader for consensus using the ZAB protocol (ZooKeeper Atomic Broadcast).

3. Consistency Guarantees:

   • Sequential Consistency: Updates are applied in the order they are received.

   • Atomicity: Updates are all-or-nothing.

   • Reliability: Once applied, updates persist.

   • Timeliness: Clients eventually see the latest data.

4. Hierarchical Namespace:

   • Data in ZooKeeper is organized into a hierarchical namespace similar to a filesystem, where each node in the tree is a znode.

---

What is a Znode?

A znode is the fundamental data structure in ZooKeeper, representing a node in the hierarchical namespace.

Types of Znodes:

1. Persistent Znode:

   • Remains in the ZooKeeper tree until explicitly deleted.

   • Suitable for storing metadata or configuration information.

2. Ephemeral Znode:

   • Exists only as long as the session that created it is active.

   • Used for temporary metadata, such as leader election or distributed locks.

3. Sequential Znode:

   • Automatically appends a monotonically increasing counter to the znode name when created.

   • Useful for ordering, such as in leader election or generating unique IDs.

4. Container Znode:

   • Similar to persistent znodes but optimized for ephemeral children.

   • Automatically deleted when the last child is removed.

---

Key Znode Operations

1. Create:

   • Create a znode with specific data.

   • Example: create /myapp/config {"key": "value"}.

2. Read:

   • Fetch data or metadata of a znode.

   • Example: get /myapp/config.

3. Update:

   • Modify the data stored in a znode.

   • Example: set /myapp/config {"key": "newValue"}.

4. Delete:

   • Remove a znode from the hierarchy.

   • Example: delete /myapp/config.

5. Watch:

   • Attach a watcher to a znode to monitor changes (data updates, children additions/deletions).

   • Example: watch /myapp/config.

---

Use Cases of Znodes

1. Configuration Management:

   • Store and manage distributed application configurations.

2. Service Discovery:

   • Register service metadata in znodes for clients to locate services dynamically.

3. Leader Election:

   • Use ephemeral sequential znodes to determine the leader in a distributed system.

4. Distributed Locking:

   • Implement locks using ephemeral znodes to synchronize access to shared resources.

5. Queue Management:

   • Use sequential znodes to build task queues with ordered processing.

     /myapp
       ├── /myapp/config          (Persistent Znode for configuration)
       ├── /myapp/leader          (Ephemeral Znode for leader election)
       └── /myapp/tasks           (Parent node for task queue)
             ├── /myapp/tasks/task0001 (Sequential Znode for task 1)
             └── /myapp/tasks/task0002 (Sequential Znode for task 2)
     

     Advantages of Znodes

     • Lightweight and fast operations.

     • Facilitates synchronization and coordination.

     • Flexible hierarchy for dynamic use cases.

     Limitations

     • Znode size is limited to 1MB.

     • Performance degrades with too many znodes in a single parent.

     ZooKeeper and its znode functionality are crucial for building reliable and fault-tolerant distributed systems.