I deployed the secondary namenode on the same machine is my main namenode:
This is wrong for performance and durability reasons (the secondary name node isn't a hot spare, but it does have a copy of needed metadata). I have found documentation on how to move the namenode, but nothing about moving the secondary namenode.
Does anyone with experience know how to do this safely?
It should be safe enough to simply remove the Secondary Namenode role, and then add it back again onto a different node on the cluster. In the intermediate period, you may see a warning from Cloudera Manager that the role doesn't exist (which may cause long term issues with the namenode), but not having a Secondary doesn't put your data any more at risk.
The job of the 2NN is to read the changes on the HDFS filesystem and add them to the
fsimage
. This decreases NN startup time as during startup the NN reads thefsimage
file and then applies all the interim log changes on top of it. The naming is somewhat unfortunate as it is indeed not a standby/backup NN but only a utility to increase NN performance.In CM there's an option to "Roll Edits" (depending on your CM version) on the 2NN, see also "Checkpointing". Make sure to do that before moving.
To be safe stop all services
Move the 2NN role to a new machine.
Restart all services
(Optional but worth it): Implement HA