Install delegate hosts

The procedures on this page will need to be repeated for each delegate host. The procedures assume that the prerequisite steps have already been completed.

Some steps will require a working master host as a prerequisite.

Preparing a delegate host

Perform the steps in Downloading and staging required files, before performing this procedure. Use this procedure to prepare a RHEL/CentOS host as a Control Center delegate host.

Log in to the candidate delegate host as root or as a user with superuser privileges.
Ensure the host has a persistent numeric ID. Each Control Center host must have a unique host ID, and the ID must be persistent (not change when the host reboots).
```
test -f /etc/hostid || genhostid ; hostid
```
Record the ID for comparison with other Control Center hosts.
Disable the firewall, if necessary. This step is required for installation but not for deployment. For more information, refer to Planning a Resource Manager deployment.
1. Determine whether the firewalld service is enabled.
```
systemctl status firewalld.service
```
  - If the result includes Active: inactive (dead), the service is disabled. Proceed to the next step.
  - If the result includes Active: active (running), the service is enabled. Perform the following substep.
2. Disable the firewalld service.
```
systemctl stop firewalld && systemctl disable firewalld
```
  On success, the preceding commands display messages similar to the following example:
```
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
rm '/etc/systemd/system/basic.target.wants/firewalld.service'
```
Optional: Enable persistent storage for log files, if desired. By default, RHEL/CentOS systems store log data only in memory or in a ring buffer in the /run/log/journal directory. By performing this step, log data persists and can be saved indefinitely, if you implement log file rotation practices. For more information, refer to your operating system documentation. Note: The following commands are safe when performed during an installation, before Docker or Control Center are installed or running. To enable persistent log files after installation, stop Control Center, stop Docker, and then enter the following commands.
```
mkdir -p /var/log/journal && systemctl restart systemd-journald
```
Enable and start the Dnsmasq package. The package facilitates networking among Docker containers.
```
systemctl enable dnsmasq && systemctl start dnsmasq
```
Most deployments do not need specific configuration for dnsmasq, however if name resolution in your environment relies solely on entries in /etc/hosts, configure dsnmasq so that containers can use the file:
1. Open /etc/dnsmasq.conf with a text editor.
2. Add the following lines to the file:
```
domain-needed
bogus-priv
local=/local/
domain=local
interface=docker0
```
3. Save the file, and then close the text editor.
4. Restart the dnsmasq service.
```
systemctl restart dnsmasq
```

Enable a higher limit for open files in containers.

Append parameters to the sysctl configuration file:

cat <<EOF >> /etc/sysctl.conf
# added for Control Center containers on [$(date)]
fs.inotify.max_user_instances=10000
fs.inotify.max_user_watches=640000
EOF

Apply the configuration file changes:
```
sysctl -p
```

Start chrony, if necessary. The chrony utility provides time synchronization, which is required by multiple Control Center internal services. You may use ntp if you prefer. Determine whether the chrony service is running.
```
systemctl status chronyd
```
If the result does not include active (running), enable and start it.
```
systemctl enable chronyd && systemctl start chronyd
```
Install the iptable_nat kernel module, if necessary.

The kernel module iptable_nat is required by the Hbase component of Resource Manager. However, it is not installed by default on all versions of RHEL 8. Follow these steps to check if it's installed and enable it, if needed.
1. Check that the module is installed:
```
lsmod | grep iptable_nat
```
  If the previous command returns nothing, the module is not installed.
2. Enable the module:
```
sudo modprobe iptable_nat
```
3. Allow the module to persist across reboots:
```
sudo echo iptable_nat > /etc/modules-load.d/iptable_nat.conf
```

Installing Docker and Control Center

Perform the steps in Downloading and staging required files before performing the procedures on this page.

Log in to the host as root or as a user with superuser privileges.
Install Docker CE 20.10.17 from the local repository mirror.
1. Install Docker CE. All hosts except RHEL 8.4+ hosts:
```
yum install --enablerepo=zenoss-mirror docker-ce-20.10.17-3.el7 docker-ce-cli
```
  RHEL 8.4+ hosts only:
```
yum install --enablerepo=zenoss-mirror docker-ce-20.10.17-3.el7 docker-ce-cli containerd.io-1.6.7-3.1.el7
```
  If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
2. Enable automatic startup.
```
systemctl enable docker
```
Install Control Center from the local repository mirror.
1. Install Control Center.
```
yum install --enablerepo=zenoss-mirror /opt/zenoss-repo-mirror/serviced-*.x86_64.rpm
```
  If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
2. Enable automatic startup.
```
systemctl enable serviced
```
Make a backup copy of the Control Center configuration file.
1. Make a copy of /etc/default/serviced.
```
cp /etc/default/serviced /etc/default/serviced-VERSION-orig
```
2. Set the backup file permissions to read-only.
```
chmod 0440 /etc/default/serviced-VERSION-orig
```
On delegate hosts only, remove unused maintenance scripts. For more information about maintenance scripts, see Control Center maintenance scripts. On all delegate hosts (never the master host), enter the following command:
```
rm -f /etc/cron.hourly/serviced  /etc/cron.weekly/serviced-fstrim
```

Configuring NFS 4.0

Use this procedure to configure NFS 4.0 on delegate hosts if the operating system release is 7.4 or a more recent release. There may be a file locking defect in NFS 4.1 with RHEL/CentOS 7.4.

Log in to the host as root or as a user with superuser privileges.
Determine which release is installed.
```
cat /etc/redhat-release
```
- If the result is 7.4 or greater, perform the remaining steps of this procedure.
- If the result includes 7.2 or 7.3, continue to the next procedure.
- Change the NFS configuration file.
- Open /etc/nfsmount.conf with a text editor.
- Locate the Defaultvers directive.
- Remove the number sign character (#) from the beginning of the line.
- Change the value from 4 to 4.0.The line should appear as follows:
```
Defaultvers=4.0
```
- Save the file, and then close the editor.
- Restart the NFS server.
```
systemctl restart nfs-server
```

Configuring Docker on delegate hosts

Use this procedure to configure Docker.

Log in to the delegate host as root or as a user with superuser privileges.
Create a symbolic link for the Docker temporary directory. Docker uses its temporary directory to spool images. The default directory is /var/lib/docker/tmp. The following command specifies the same directory that Control Center uses, /tmp. You can specify any directory that has a minimum of 10GB of unused space.
1. Create the docker directory in /var/lib.
```
mkdir /var/lib/docker
```
2. Create the link to /tmp.
```
ln -s /tmp /var/lib/docker/tmp
```

Create a systemd drop-in file for Docker.

Create the override directory.

mkdir -p /etc/systemd/system/docker.service.d

Create the unit drop-in file.

cat <<EOF > /etc/systemd/system/docker.service.d/docker.conf
[Service]
TimeoutSec=300
EOF

Reload the systemd manager configuration.
```
systemctl daemon-reload
```

Create an LVM thin pool for Docker data. For more information about the serviced-storage command, see serviced-storage.To use an entire block device or partition for the thin pool, replace Device-Path with the device path:
```
serviced-storage create-thin-pool docker Device-Path
```
On success, the result is the device mapper name of the thin pool, which always starts with /dev/mapper.
Configure and start the Docker service.
1. Create a variable for the name of the Docker thin pool.
  
  Replace Thin-Pool-Device with the name of the thin pool device created in the previous step:
```
myPool="Thin-Pool-Device"
```
2. Create the Docker daemon configuration file. The exec-opt field is a workaround for a Docker issue on RHEL/CentOS 7.x systems.
```
mkdir /etc/docker && cat <<EOF > /etc/docker/daemon.json
{
  "bip": "172.17.0.1/16",
  "dns": [
    "172.17.0.1"
  ],
  "exec-opts": [
    "native.cgroupdriver=cgroupfs"
  ],
  "log-level": "error",
  "storage-driver": "devicemapper",
  "storage-opts": [
    "dm.mountopt=discard",
    "dm.thinpooldev=$myPool"
  ]
}
EOF
```
3. Review the file to ensure it is correct.
```
cat /etc/docker/daemon.json
```
4. Start or restart Docker.
```
systemctl restart docker
```
  The startup may take up to a minute, and may fail. If startup fails, repeat the restart command.
Configure name resolution in containers. Each time it starts, docker selects an IPv4 subnet for its virtual Ethernet bridge. The selection can change; this step ensures consistency.
1. Identify the IPv4 address and netmask docker has selected for its virtual bridge.
```
ip addr show docker0 | awk '/^ *inet/ { print $2 }'
```
  If the result is not 172.17.0.1/16, open /etc/docker/daemon.json, update the dns and bip fields, and then restart the Docker service.
2. Restart the Docker service.
```
systemctl restart docker 
```
3. Confirm that there are no stale connections still using the original subnet. If any are found, Docker may require a full stop and start:
```
conntrack -F && conntrack -L
```
4. Verify there is an iptables masquerade rule for the configured docker0 subnet:
```
iptables -t nat -L -n
```

Enabling use of the command-line interface

Use this procedure to enable a user to perform administrative tasks with the Control Center command-line interface.

Log in to the host as root or as a user with superuser privileges.
Add a user to the serviced and docker groups.

Replace User with the name of a login account on the host.
```
usermod -aG serviced,docker User
```
Repeat the preceding command for each user to add.

Setting the host role to delegate

Use this procedure to configure a host as a delegate host. Perform these steps:

Log in to the host as root or as a user with superuser privileges.
Edit the Control Center configuration file.
1. Open /etc/default/serviced in a text editor.
2. Locate the line for the SERVICED_MASTER variable, and then make a copy of the line, immediately below the original.
3. Remove the number sign character (#) from the beginning of the line.
4. Change the value from 1 to 0.
5. Save the file, and then close the editor.

Verify the settings in the serviced configuration file.

grep -E '^[[:space:]]*[A-Z_]+' /etc/default/serviced

Setting the Docker registry endpoint

Use this procedure to configure the delegate host with the endpoint of the Docker registry for Control Center. Unless the master host is configured with an alternative local Docker registry, which is rare, the endpoint is the master host's hostname or IP address and port 5000.

Perform these steps:

Log in to the delegate host as root or as a user with superuser privileges.
Edit the Control Center configuration file.
1. Open /etc/default/serviced in a text editor.
2. Locate the line for the SERVICED_DOCKER_REGISTRY variable, and then make a copy of the line, immediately below the original.
3. Remove the number sign character (#) from the beginning of the line.
4. Replace localhost:5000 with the endpoint of the local Docker registry. If the master host is configured with an alternative local Docker registry, use the same endpoint here. Otherwise, just replace localhost with the IP address or fully-qualified domain name of the Control Center master host.
5. Save the file, and then close the editor.

Verify the settings in the serviced configuration file.

grep -E '^[[:space:]]*[A-Z_]+' /etc/default/serviced

Add the insecure registry flag to the Docker configuration file.
1. Open /etc/docker/daemon.json in a text editor.
2. Add the local Docker registry endpoint to the file. Replace Registry-Endpoint with the same value used for the SERVICED_DOCKER_REGISTRY variable:
```
{
  "insecure-registries" : ["Registry-Endpoint"]
}
```
  Note
  
  The option above assumes an otherwise blank file. If there are other options already in place, the insecure-registries option should be placed at the correct level. See the Docker documentation for more information.
3. Save the file, and then close the editor.
Restart the Docker service.
```
systemctl restart docker
```

Setting internal services endpoints

Use this procedure to configure a delegate host with the endpoints of the Control Center internal services.

The following variables identify the internal services endpoints:

Perform these steps:

Log in to the delegate host as root or as a user with superuser privileges.
Edit the Control Center configuration file.
1. Open /etc/default/serviced in a text editor.
2. For each endpoint variable, locate the line that sets the variable, and then make a copy of the line, immediately below the original.
3. Remove the number sign character (#) from the beginning of the line.
4. Replace localhost or {{SERVICED_MASTER_IP}} with the IP address or hostname of the master host.
5. Save the file, and then close the editor.

Verify the settings in the serviced configuration file.

grep -E '^[[:space:]]*[A-Z_]+' /etc/default/serviced

Optional procedures

The following procedures may be required for your installation. Perform these procedures only if you are sure they are necessary for your use case.

Configuring offline use

Use this procedure to configure a host to operate without internet access.

Perform these steps:

Log in to the host as root or as a user with superuser privileges.
Identify the IPv4 address of the host.
```
hostname -i
```
Edit the Control Center configuration file.
1. Open /etc/default/serviced in a text editor.
2. Locate the line for the SERVICED_OUTBOUND_IP variable, and then make a copy of the line, immediately below the original.
3. Remove the number sign character (#) from the beginning of the line.
4. Change the value to the IPv4 address identified in the previous step.
5. Save the file, and then close the editor.
Verify the settings in the serviced configuration file.
```
grep -E '^\b*[A-Z_]+' /etc/default/serviced
```

Installing a security certificate

The default, insecure certificate that Control Center uses for TLS-encrypted communications are based on a public certificate compiled into serviced. Use this procedure to replace the default certificate files with your own files.

If you are using virtual host public endpoints for your Zenoss Service Dynamics deployment, you need a wildcard certificate or a subject alternative name (SAN) certificate.
If your end users access the browser interface through a reverse proxy, the reverse proxy may provide the browser with its own SSL certificate. If so, please contact Zenoss Support for additional assistance.

To perform this procedure, you need valid certificate files. For information about generating a self-signed certificate, see Creating a self-signed security certificate.

To use your own certificate files, perform this procedure on the Control Center master host and on each Control Center delegate host in your environment.

Follow these steps:

Log in to the host as root or as a user with superuser privileges.
Use a secure copy program to copy the key and certificate files to /tmp.
Move the key file to the /etc/pki/tls/private directory. Replace <KEY_FILE> with the name of your key file:
```
mv /tmp/<KEY_FILE>.key /etc/pki/tls/private
```
Move the certificate file to the /etc/pki/tls/certs directory. Replace <CERT_FILE> with the name of your certificate file:
```
mv /tmp/<CERT_FILE>.crt /etc/pki/tls/certs
```
Updates only: Create a backup copy of the Control Center configuration file. Do not perform this step for a fresh install:
```
cp /etc/default/serviced /etc/default/serviced.before-cert-files
```
Edit the Control Center configuration file.
1. Open /etc/default/serviced in a text editor.
2. Locate the line for the SERVICED_KEY_FILE variable, and then make a copy of the line, immediately below the original.
3. Remove the number sign character (#) from the beginning of the line.
4. Replace the default value with the full pathname of your key file.
5. Locate the line for the SERVICED_CERT_FILE variable, and then make a copy of the line, immediately below the original.
6. Remove the number sign character (#) from the beginning of the line.
7. Replace the default value with the full pathname of your certificate file.
8. Save the file, and then close the editor.

Verify the settings in the configuration file.

grep -E '^[[:space:]]*[A-Z_]+' /etc/default/serviced

Updates only: Reload the systemd manager configuration. Do not perform this step for a fresh install:
```
systemctl daemon-reload
```

Setting Zookeeper security variables

Use this procedure to add security to the Zookeeper instances for Control Center and Resource Manager. For more information, see ZooKeeper security.

Log in to the Control Center host as root or as a user with superuser privileges.
Open /etc/default/serviced in a text editor.
Add a user account name to secure the leader-election phase.
1. Locate the line for the SERVICED_ISVCS_ZOOKEEPER_USERNAME variable, and then make a copy of the line, immediately below the original.
2. Remove the number sign character (#) from the beginning of the line.
3. Add a value for the variable.
Add a password to secure the leader-election phase.
1. Locate the line for the SERVICED_ISVCS_ZOOKEEPER_PASSWD variable, and then make a copy of the line, immediately below the original.
2. Remove the number sign character (#) from the beginning of the line.
3. Add a value for the variable.
Add a user account name to secure data nodes.
1. Locate the line for the SERVICED_ZOOKEEPER_ACL_USER variable, and then make a copy of the line, immediately below the original.
2. Remove the number sign character (#) from the beginning of the line.
3. Add a value for the variable.
Add a password to secure data nodes.
1. Locate the line for the SERVICED_ZOOKEEPER_ACL_PASSWD variable, and then make a copy of the line, immediately below the original.
2. Remove the number sign character (#) from the beginning of the line.
3. Add a value for the variable.
Save the file, and then close the editor.

Verify the settings.

grep -E '^[[:space:]]*[A-Z_]+' /etc/default/serviced

Additional features

You may wish to adjust other Control Center settings or behaviors before starting Control Center for the first time. If so, please refer to the delegate host and universal configuration variables pages.

Starting Control Center on delegate hosts

Use this procedure to start serviced on a delegate host for the first time.

Log in to the delegate host as root or as a user with superuser privileges.

Verify the settings in the serviced configuration file.

grep -E '^[[:space:]]*[A-Z_]+' /etc/default/serviced

Start the Control Center service (serviced).
```
systemctl start serviced
```
To monitor progress, enter the following command:
```
journalctl -flu serviced -o cat
```

Authenticating delegate hosts

Control Center uses RSA key pairs to create the authentication tokens that are required for all delegate communications. When you add a host to a resource pool, the serviced instance on the master host creates a private key for the delegate and bundles it with its own public key. The serviced instance on the delegate host uses the bundle to sign messages with its unique tokens.

Key bundles are installed by using an SSH connection or a file.

The command to add a host to a pool can initiate an SSH connection with the delegate and install the key bundle. This option is the most secure, because no file is created. However, it requires either public key authentication or password authentication between the master and delegate hosts.
When no SSH connection is requested, the command to add a host to a pool creates a file containing the key bundle. You can move the key bundle file to the delegate host with any file transfer method, and then install it on the delegate.

The following procedures demonstrate how to add a host to a resource pool and install its key bundle.

Adding a delegate host to a pool through SSH

To succeed, the following statements about the login account used to perform this procedure must be true:

The account exists on both the master host and on the delegate host.
The account has serviced CLI privileges.
The account has either public key authentication or password authentication enabled on the master host and on the delegate host.

Use this procedure to add a delegate host to a resource pool through an SSH connection.

Log in to the Control Center master host as a user with serviced CLI privileges.
Add a delegate host to a resource pool.

If the master and delegate host are configured for key-based access, the following command does not prompt you to add the delegate to the list of known hosts or to provide the password of the remote user account.

Use a hostname or IP address to identify the delegate. If you use a hostname, all Control Center hosts must be able to resolve it, either through an entry in /etc/hosts or through a nameserver on the network.

In the following example, replace Hostname-Or-IP with the hostname or IP address of a delegate host, and replace Resource-Pool with the name of a resource pool.

If the host is behind a router or firewall for network address translation (NAT), include the --nat-address option to specify the NAT device's hostname or IP address and port of the delegate host.
```
serviced host add --register Hostname-Or-IP:4979 Resource-Pool [--nat-address==NAT-Hostname-Or-IP:NAT-Port]
```

Adding a delegate to a pool with a file

Use this procedure to add a delegate host to a resource pool with a key bundle file.

Log in to the Control Center master host as a user with serviced CLI privileges.
Add a delegate host to a resource pool.

Use a hostname or IP address to identify the delegate. If you use a hostname, all Control Center hosts must be able to resolve it, either through an entry in /etc/hosts or through a nameserver on the network.

In the following example, replace Hostname-Or-IP with the hostname or IP address of a delegate host, and replace Resource-Pool with the name of a resource pool.

If the host is behind a router or firewall for network address translation (NAT), include the --nat-address option to specify the NAT device's hostname or IP address and port of the delegate host.
```
serviced host add Hostname-Or-IP:4979 Resource-Pool [--nat-address==NAT-Hostname-Or-IP:NAT-Port]
```
The command creates a unique key bundle file in the local directory.
Use a file transfer utility such as scp to copy the key bundle file to the delegate host. Once copied to the delegate host, the key bundle file is not needed on the master host and can be deleted.
Log in to the Control Center delegate host as a user with serviced CLI privileges.
Install the key bundle. Replace Key-Bundle-Path with the pathname of the key bundle file:
```
serviced host register Key-Bundle-Path
```
Delete the key bundle file. The file is no longer needed on the delegate host. Replace Key-Bundle-Path with the pathname of the key bundle file:
```
rm Key-Bundle-Path
```