Redundant management nodes in MySQL Cluster

Every time I teach the MySQL Cluster architecture, someone inevitably asks "Isn't the management node (ndb_mgmd) a single point of failure?" The short answer: no. The management node is not a SPOF because the cluster can continue without it. However, it's inconvenient if your management node is down because the management node does several things such as:

• Provide status information about the cluster and allow you to use the ndb_mgm for various maintenance tasks like taking a hot backup
  
• Own the cluster config file (therefore it must be running to start a node)
  
• Arbitration in case of a potential split-brain
  
• Logging
  

So while the management node can be down, it is nice to have a redundant one for failover. This is very easy to do:

1. Add 2 [NDB_MGMD] sections to config.ini:
   [NDB_MGMD]
   #Id is required when defining multiple mgmt nodes
   Id=1
   Hostname=192.168.0.31
   
   [NDB_MGMD]
   Id=2
   Hostname=192.168.0.32
   
   
2. Change the ndb-connectstring to include both IPs of the management nodes:
   [mysql_cluster]
   ndb-connectstring=192.168.0.31,192.168.0.32
   
   
3. Make sure the config.ini is on both management nodes and that the files are identical. Start both ndb_mgmd nodes.
   

That's it! The management nodes will act in an active-passive way and failover as necessary. Make sure you do not run any management node on the same physical host as a data node - it will cause a cluster shutdown if they fail simultaneously.