Lets learn "How to replace a failed etcd member"

-

 


It is possible that one of the nodes in the etcd cluster went into a bad state either because of network issues or due to disk corruption. In such scenarios, one requires replacing an etcd member in etcd cluster. This tutorial explains the steps to replace etcd member. This tutorial assumes that the reader has sufficient knowledge about what exactly etcd is.

The terminology used in this tutorial:

  • Peer node: One of the active nodes in etcd cluster
  • Corrupted node: Node which is going to be replaced
  • Fresh node: Node with which the corrupted node is going to be replaced with.

So below is the list of commands we need to run to perform the etcd replacement.
  1. Stop etcd service on Fresh node.
  2. Clear the etcd data directory on Fresh node just to ensure that we don't have any stale data there. The command is "rm -rf <etcd-data>" directory.
  3. The next step is we need to figure out the member id of the corrupted node. Then command to figure that out is "etcdctl member list | awk '/<Corrupted node IP>/'". This command need to be run on Peer node and this will return the output in the form of "<memberID>: name=<> peerURLs=<> clientURLs=<> isLeader=<>"
  4. Once we figure out the member id of the Corrupted node, we need to remove that member by running the following command on Peer node "etcdctl member remove <Corrupted node memberID>".
  5. The next step is to add the fresh member. To do that, we need to run "etcdctl member add <Fresh node hostname> http://<Fresh node IP>:<etcd HTTP port>" on Peer node.
  6. Now to want etcd on Fresh node to join the existing cluster, we need to temporarily modify etcd startup configuration on Fresh node using "sed -i -E 's/initial-cluster-state=new/initial-cluster-state=existing/ <etcd-config-file>".
  7. Start the etcd service on Fresh node.
  8. Verify if a member gets added using the "etcdctl member list" command on Fresh node.
  9. Revert the configuration change made in Step 6.
Hope this helps, Let me know if you have any questions in the comments below. Thanks for your time :)

Comments

Post a Comment

Popular posts from this blog

Lets learn "About kube proxy in iptables mode"

-
Image
In this tutorial, we'll learn about the Kube-proxy in iptables mode. Basically, we'll see how Kube-proxy create iptables that help in load balancing and service discovery. If you are not familiar with Linux IPTables I would recommend to check out my tutorial on IPTables here  first and then come back to read this tutorial. Kube-proxy Kube-proxy when running in iptables mode basically watch kube-apiserver for new endpoints and create corresponding iptables rules and iptables are fully responsible for service discover and proper routing of traffic to corresponding pods. Kube-proxy needs to run on all k8s worker nodes. Below is a small video describing the working of kube-proxy in iptables mode.  src:  https://www.slideshare.net/CJCullen/kubernetes-networking-55835829 Now that we know what Kube-proxy is and how it Kube-proxy works, let deep dive into the types of iptables Kube-proxy adds to the firewall. Kube-proxy added IPTables To learn this let's consider an example. Consid
Read more

Lets learn "System design for paste bin (or any text sharing website)"

-
Image
  In this tutorial, we'll learn about the system design for PasteBin. If you don't know what Pastebin is here goes its definition. Pastebin is a type of online content hosting service where users can store texts to share with others over the internet.  Pastebin system design requirements  Functional Basic - User should be able to paste the text, generate the link and share the link with other users. There should be the max size of the text that our system will support (say 10Mb) Users should be able to provide the custom URL path for the link created. Like if a user wants to create the links  www.pb.com/letlearn, he can give this URL and pastebin should create a link using this URL to share with others. The link or paste expiry feature should be there for each stored text. There can be default expiry time say 1 year but this should be configurable. Personalization means the user should be able to login to track all the pastes created by him and manage them. Non-Functional Durab
Read more

Lets learn "Factory design pattern"

-
 In this tutorial, we'll learn about the factory design pattern as in what it is, in which use cases this is being used and how it works. If you've are not familiar with what are software design patterns and what are its different types check out this tutorial. Introduction A factory design pattern is a creational design pattern that is related to object creation. In this pattern, we'll have a method to create an object without exposing its creation logic to the client. If a class method is creating an object it has to be static.  The idea is to create a static method ( factory method ) which instantiates a class without exposing its logic to the client and the client can use that method every time it needs to create an object of the class. The reason why the factory pattern is useful is that as it does not expose the logic of object creation, so if logic is changed, the client code need not to be changed. Usecases The place where this pattern is mostly used is library cod
Read more