HBase healthcheck failures

You may find that when starting Resource Manager that the HBase HMaster and Region Server services do not start, or they may start but fail their healthchecks. The same behavior may be seen on the Opentsdb Reader, OpenTSDB Writer, and Metrics CentralQuery services.

In this case, the HMaster service container logs may contain lines like the following:

Service All ZooKeepers up not starting. Output: Ncat: Could not resolve hostname "zk1": Name or service not known. QUITTING.

This may be caused by the absence of the iptable_nat kernel module.

Explanation

Certain versions of Red Hat Enterprise Linux 8 (RHEL8) do not ship with the iptable_nat module enabled. If your Resource Manager instance is newly deployed on one of these versions, or if you have upgraded the OS of a previously working instance, you may encounter this issue.

Procedure

Use the following steps to install the iptable_nat module, if necessary.

Check that the module is installed:
```
lsmod | grep iptable_nat
```
If the previous command returns nothing, the module is not installed.
Enable the module:
```
sudo modprobe iptable_nat
```

Allow the module to persist across reboots:

sudo echo iptable_nat > /etc/modules-load.d/iptable_nat.conf