Updating Control Center to release 2.0.2

This section includes procedures for updating Control Center 1.11.0 to release 2.0.2.

The following list outlines recommended best practices for updating Control Center deployments:

Review the release notes for this release and relevant prior releases. The latest information is provided there.
On delegate hosts, most of the update steps are identical. Use screen, tmux, or a similar program to establish sessions on each delegate host and perform the steps at the same time.
Review and verify the settings in delegate host configuration files (/etc/default/serviced) before starting the update. Ideally, the settings on all delegate hosts are identical, except on ZooKeeper nodes and delegate hosts that don't mount the DFS.
Review the update procedures before performing them. Every effort is made to avoid mistakes and anticipate needs; nevertheless, the instructions may be incorrect or inadequate for some requirements or environments.

Note

The supported update path is from Control Center 1.11.0 to Control Center 2.0.2. If you're running a Control Center version older than 1.11.0, update to 1.11.0 first.

Release 2.0.2 introduces three steps that earlier Control Center updates didn't require:

Backing up the deployment: There's no way to return to release 1.11.0 after the update starts. Take a backup first.
Clearing the internal services HBase data: The update clears the embedded HBase data that stores Control Center internal metrics.
Configuring Docker registry authentication: The internal Docker registry now requires authentication. You create a registry password file on the master host, and you log in to the registry on every host in the deployment.

Zenoss Service Dynamics deployments install the serviced-zsd package. The serviced package contains the same Control Center but is compiled for a newer Linux release, and it doesn't start on a Zenoss Service Dynamics host.

You can't return to release 1.11.0 after the update starts

Release 2.0.2 migrates the internal Docker registry forward and records it as migrated, and the internal data stores move forward with it. There's no downgrade path. If you have to return to release 1.11.0, the only route is to restore a backup that you took before the update. See Backing up the deployment.

Historical Control Center metrics are deleted

Clearing the embedded HBase data deletes the metrics that Control Center collected about its own internal services. Graphs for internal services start from an empty data set after the update, and metrics collected before the update can't be recovered. Application data isn't affected.

Updating 1.11.0 to 2.0.2

The update has seven phases. Complete them in order.

Back up the deployment
Download the required files
Stage Docker image files on the master host
Update the master host
Update the delegate hosts
Start Control Center
Perform post-update procedures

Backing up the deployment

Take a backup before you begin. This step is required, not optional. Once the update starts, there's no way to return to release 1.11.0 except by restoring a backup taken beforehand.

Log in to the master host as a user with serviced CLI privileges.
Confirm that enough space is available in the backup file system. For sizing guidance, see Back up and restore.

Create the backup.

serviced backup /opt/serviced/var/backups

Confirm that the backup completed and note its name. You need the name if you have to restore.
```
serviced backup ls
```

If you need to return to release 1.11.0 after a failed update, reinstall release 1.11.0 and then restore this backup. For the restore procedure, see Restoring.

Downloading Control Center files for release 2.0.2

Use this procedure to download required files to a workstation, and then copy the files to the hosts that need them.

To perform this procedure, you need:

Internet access: A workstation that can reach the download site.
Permission: Access to download files from delivery.zenoss.io. Customers can request permission by filing a ticket at the Zenoss Support site.
A secure network copy program: Used to copy files to Control Center hosts.

Follow these steps:

In a web browser, navigate to delivery.zenoss.io, and then log in.
Download the self-installing Docker image files.
- install-zenoss-serviced-isvcs_v76.run
- install-zenoss-isvcs-zookeeper_v16.run
Download the Control Center RPM file. Zenoss Service Dynamics deployments require the serviced-zsd package. Download this exact file name.
- serviced-zsd-2.0.2-1.x86_64.rpm
Don't download the serviced package by mistake

The download site also offers serviced-2.0.2-1.x86_64.rpm. It contains the same Control Center but is compiled for a newer Linux release. On a Zenoss Service Dynamics host it installs successfully and then fails to start, which makes the mistake hard to recognize. Check the file name before you download.
Identify the operating system release on Control Center hosts. All Control Center hosts should be running the same operating system release and kernel.
```
cat /etc/redhat-release
```
Download the RHEL/CentOS repository mirror file. The download site provides a repository mirror file containing the packages that Control Center requires. Both RHEL 8 and RHEL 9 use the same mirror file.
- <<MIRROR-RPM>>
Optional: Download the Zenoss Pretty Good Privacy (PGP) key. You can use the key to verify RPM files and the yum metadata of the repository mirror.
1. Download the key.
```
curl --location -o /tmp/tmp.html 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xED0A5FD2AA5A1AD7'
```
2. Determine whether the download succeeded. If the result is 0, repeat the previous substep. If the result is 1, continue.
```
grep -Ec '^\-\-\-\-\-BEGIN PGP' /tmp/tmp.html
```
3. Extract the key.
```
awk '/^-----BEGIN PGP.*$/,/^-----END PGP.*$/' /tmp/tmp.html > ./RPM-PGP-KEY-Zenoss
```
Use a secure copy program to copy the files to Control Center hosts. Copy all files to the master host. Copy the repository mirror file, the Control Center RPM file, and the PGP key file to all delegate hosts. Copy the ZooKeeper image file to delegate hosts that are ZooKeeper ensemble nodes.

Staging Docker image files on the master host

Before performing this procedure, verify that sufficient temporary space is available on the file system where /root is located. The serviced-isvcs:v76 image is approximately 2.2 GB.

Use this procedure to copy Docker image files to the Control Center master host.

Log in to the master host as root or as a user with superuser privileges.
Copy or move the archive files to /root.
Add execute permission to the files.
```
chmod +x /root/*.run
```

Staging a Docker image file on ZooKeeper ensemble nodes

Before performing this procedure, verify that approximately 170 MB of temporary space is available on the file system where /root is located.

Use this procedure to add a Docker image file to the Control Center delegate hosts that are ZooKeeper ensemble nodes. Delegate hosts that aren't ZooKeeper ensemble nodes don't need the file.

Log in to a delegate host as root or as a user with superuser privileges.
Copy or move the install-zenoss-isvcs-zookeeper_v*.run file to /root.
Add execute permission to the file.
```
chmod +x /root/*.run
```

Update the master host

Perform the following procedures on the master host of a multi-host deployment, or on the single host in a single-host deployment. Procedures for delegate hosts are in Update the delegate hosts.

Installing the repository mirror

Use this procedure to install the Zenoss repository mirror on a Control Center host. The mirror contains packages that all Control Center hosts require. Repeat this procedure on each host in your deployment.

Log in to the target host as root or as a user with superuser privileges.
Move the RPM files and the PGP key file to /tmp.
Updates only: Remove the existing repository mirror, if necessary.
1. Search for the mirror.
```
yum list --disablerepo=* | awk '/^yum-mirror/ { print $1}'
```
2. Remove the mirror. Replace Old-Mirror with the name that the previous substep returned.
```
yum remove Old-Mirror
```
Install the repository mirror. The yum command copies the contents of the RPM file to /opt/zenoss-repo-mirror.
```
yum install /tmp/yum-mirror-*.rpm
```
Optional: Install the PGP key, and then test the package files.
1. Move the key to the mirror directory.
```
mv /tmp/RPM-PGP-KEY-Zenoss /opt/zenoss-repo-mirror
```
2. Install the key.
```
rpm --import /opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
```
3. Test the repository mirror package file. On success, the result includes the file name and (sha1) dsa sha1 md5 gpg OK.
```
rpm -K /tmp/yum-mirror-*.rpm
```
4. Test the Control Center package file.
```
rpm -K /tmp/serviced-zsd-2.0.2-1.x86_64.rpm
```

Move the Control Center package file to the mirror directory.

mv /tmp/serviced-zsd-2.0.2-1.x86_64.rpm /opt/zenoss-repo-mirror

Stopping Control Center on the master host

Use this procedure to stop the Control Center service (serviced) on the master host.

Log in to the master host as root or as a user with superuser privileges.
Stop the top-level service that serviced manages, if necessary.
1. Show the status of running services. The top-level service is listed immediately below the headings line. If the top-level service and all child services are stopped, go to step 3.
```
serviced service status
```
2. Stop the top-level service.
```
serviced service stop Zenoss.resmgr
```
3. Monitor the stop. When the top-level service and all child services are stopped, go to the next step.
```
serviced service status
```
Stop the Control Center service.
```
systemctl stop serviced
```
Confirm that the service is stopped. The result is inactive.
```
systemctl is-active serviced
```
Ensure that no containers remain in the local repository.
1. Display the identifiers of all containers, running and exited. If the command returns no result, this procedure is complete.
```
docker ps -qa
```
2. Remove all remaining containers.
```
docker ps -qa | xargs --no-run-if-empty docker rm -fv
```
3. Display the identifiers of all containers again. If the command returns no result, this procedure is complete. Otherwise, disable automatic startup of serviced, reboot the host, log in again, and re-enable automatic startup.
```
docker ps -qa
```

Loading image files

Use this procedure to load images into the local Docker registry on a host.

Log in to the host as root or as a user with superuser privileges.
Change directory to /root.
```
cd /root
```

Load the images.

for image in install-zenoss-*.run
do
  /bin/echo -en "\nLoading $image..."
  yes | ./$image
done

List the images in the registry. The result shows one image for each archive file.
```
docker images
```

Updating the serviced binary for release 2.0.2

Use this procedure to update the serviced binary on a Control Center host. Perform this procedure on each host in your Control Center deployment. In multi-host deployments, stop serviced on the master host first.

Log in to the host as root or as a user with superuser privileges.

Start the docker service, if necessary.

systemctl is-active docker || systemctl start docker

Disable SELinux temporarily, if necessary.
1. Determine the current mode. If the result isn't enforcing, go to step 4.
```
sestatus | awk '/Current mode:/ { print $3 }'
```
2. Disable SELinux temporarily.
```
setenforce 0
```

Save the current serviced configuration file as a reference and set permissions to read-only.

mv /etc/default/serviced /etc/default/serviced-pre-2.0.2
chmod 0440 /etc/default/serviced-pre-2.0.2

Install the new Control Center package. If yum returns a dependency error, see Resolving package dependency conflicts.
```
yum install --enablerepo=zenoss-mirror /opt/zenoss-repo-mirror/serviced-zsd-2.0.2-1.x86_64.rpm
```
Enable SELinux, if you disabled it in step 3.
```
setenforce 1
```

Make a backup copy of the new configuration file and set permissions to read-only.

cp /etc/default/serviced /etc/default/serviced-2.0.2-orig
chmod 0440 /etc/default/serviced-2.0.2-orig

Compare the new configuration file with the configuration file of the previous release.
```
diff /etc/default/serviced-1.11.0-orig /etc/default/serviced-2.0.2-orig
```
Restore the backup of the previous configuration file. If the previous command returned differences, use them to edit the restored file. For more information, see Configuration variables.
```
cp /etc/default/serviced-pre-2.0.2 /etc/default/serviced && chmod 0644 /etc/default/serviced
```
Reload the systemd manager configuration.
```
systemctl daemon-reload
```

Note

The serviced and serviced-zsd packages replace each other, so installing one removes the other. A Zenoss Service Dynamics host only ever has serviced-zsd, so this matters only if the wrong package is installed by hand.

Configuring Docker registry authentication on the master host

Beginning with release 2.0.2, the internal Docker registry requires a user name and password. Earlier releases exposed the Docker Registry HTTP API v2 without authentication.

TLS is optional

Authentication is required, but encryption isn't. By default the registry accepts basic authentication over plain HTTP, and you don't need a certificate to complete this update. To encrypt registry traffic, set the SERVICED_DOCKER_CERT_FILE and SERVICED_DOCKER_KEY_FILE variables.

Use this procedure to create the registry password file on the master host. Perform it while serviced is stopped, after you update the Control Center binary and before you start Control Center.

Log in to the master host as root or as a user with superuser privileges.
Confirm that serviced is stopped. The result is inactive.
```
systemctl is-active serviced
```
Install the htpasswd tool, if it isn't already installed.
```
dnf install -y httpd-tools
```
Create the registry password file. Replace <registry-user> with a user name of your choice and <password> with a password of your choice.
```
htpasswd -bcBC 10 /opt/serviced/isvcs/resources/.htpasswd <registry-user> <password>
```

Set the permissions of the password file.

chmod 644 /opt/serviced/isvcs/resources/.htpasswd

Verify that the file exists and is readable.

ls -l /opt/serviced/isvcs/resources/.htpasswd

Note

Record the credentials in your organization's secret store. You need them on every host in the deployment, and again when you verify the update.

Delegate hosts don't need a copy of this file. The registry runs on the master host, so the password file is required there only. Delegate hosts need a login, which is the next procedure.

Logging in to the Docker registry

Control Center reads registry credentials from the credential file that belongs to the root user on the host where Control Center is running. Docker writes that file when you run docker login, and it doesn't copy the file to other hosts. Every host in the deployment therefore needs its own login.

A host that isn't logged in fails silently

Control Center doesn't check for credentials before it tries to download an image, and it doesn't report a missing login. The registry refuses the request, and the operator sees only an image download that fails without a stated reason. If images fail to download after you enable authentication, confirm that the affected host has been logged in.

Perform this procedure on the master host and on every delegate host.

Log in to the host as root. Run docker login as root rather than through sudo from another account, because Control Center reads the credential file belonging to root.

Identify the registry endpoint.

grep -E '^[[:space:]]*SERVICED_DOCKER_REGISTRY=' /etc/default/serviced

Log in to the registry, using the address exactly as the previous step returned it. Replace <registry-user> with the user name you set on the master host, and enter the password when prompted. The result is Login Succeeded.
```
docker login -u <registry-user> <registry-address>
```
The address has to match exactly

Docker matches a saved login to a registry by the exact text of the address, so a login saved under one form of the address doesn't apply to another. Use the value of SERVICED_DOCKER_REGISTRY as written on that host. On a delegate, that value points at the master host. Don't substitute localhost, which on a delegate refers to the delegate itself, where no registry is running.
Confirm that the credential file was written.
```
ls -l /root/.docker/config.json
```

Clearing the internal services HBase data

Release 2.0.2 upgrades HBase from version 0.99.2 to version 2.6.4. The data directory doesn't move, and nothing migrates its contents, so the new version can't read data written by the old one. Every deployment with existing internal metrics has to clear that data before starting Control Center. Use this procedure on the master host.

This deletes Control Center internal metric history

Removing the HBase data directory deletes the metrics that Control Center collected about its own internal services. After the update, graphs for internal services start from an empty data set, and earlier metrics can't be recovered. Application data isn't affected.

Log in to the master host as root or as a user with superuser privileges.
Confirm that serviced is stopped. The result is inactive. Don't continue until it is.
```
systemctl is-active serviced
```

Remove the embedded HBase data.

rm -rf /opt/serviced/var/isvcs/opentsdb/hbase/*

Confirm that the directory is empty.

ls -la /opt/serviced/var/isvcs/opentsdb/hbase/

Setting the internal services OpenTSDB credentials

Control Center requires a user name and password to run the internal services OpenTSDB database. If these credentials are already set on this host, skip this procedure.

Log in to the master host as root or as a user with superuser privileges.
Edit the Control Center configuration file.
1. Open /etc/default/serviced in a text editor.
2. Locate the line for the SERVICED_ISVCS_OPENTSDB_USERNAME variable, and then copy the line immediately below the original.
3. Remove the number sign character (#) from the beginning of the copied line.
4. Enter a user name for the OpenTSDB server. You can choose any user name.
5. Locate the line for the SERVICED_ISVCS_OPENTSDB_PASSWD variable, and then copy the line immediately below the original.
6. Remove the number sign character (#) from the beginning of the copied line.
7. Enter a password for the OpenTSDB server. You can choose any password.
8. Save the file, and then close the editor.

Verify the credentials in the configuration file.

grep -E '^[[:space:]]*SERVICED_ISVCS_OPENTSDB' /etc/default/serviced

Optional: Installing a security certificate

The default certificate that Control Center uses for TLS-encrypted communications is compiled into serviced. Use this procedure to replace the default certificate files with your own files.

If you use virtual host public endpoints, you need a wildcard certificate or a subject alternative name (SAN) certificate. If your users reach the browser interface through a reverse proxy, the proxy may present its own certificate; contact Support for assistance.

Perform this procedure on the master host and on each delegate host.

Log in to the host as root or as a user with superuser privileges.
Use a secure copy program to copy the key and certificate files to /tmp.
Move the key file to the /etc/pki/tls/private directory. Replace <KEY_FILE> with the name of your key file.
```
mv /tmp/<KEY_FILE>.key /etc/pki/tls/private
```
Move the certificate file to the /etc/pki/tls/certs directory. Replace <CERT_FILE> with the name of your certificate file.
```
mv /tmp/<CERT_FILE>.crt /etc/pki/tls/certs
```

Create a backup copy of the Control Center configuration file.

cp /etc/default/serviced /etc/default/serviced.before-cert-files

Edit the Control Center configuration file to set SERVICED_KEY_FILE and SERVICED_CERT_FILE to the full path names of your files. Uncomment each line by removing the number sign character (#).

Verify the settings in the configuration file.

grep -E '^[[:space:]]*[A-Z_]+' /etc/default/serviced

Reload the systemd manager configuration.
```
systemctl daemon-reload
```

Update the delegate hosts

Perform the following procedures on each delegate host in a multi-host deployment. Before you begin, stop Control Center on the master host.

Installing the repository mirror on a delegate host

Follow Installing the repository mirror. The procedure is identical on delegate hosts.

Stopping Control Center on a delegate host

Use this procedure to stop the Control Center service on a delegate host. Repeat it on each delegate host. Stop Control Center on the master host first.

Log in to the delegate host as root or as a user with superuser privileges.
Stop the Control Center service.
```
systemctl stop serviced
```
Ensure that no containers remain in the local repository.
1. Display the identifiers of all containers, running and exited. If the command returns no result, go to step 4.
```
docker ps -qa
```
2. Remove all remaining containers.
```
docker ps -qa | xargs --no-run-if-empty docker rm -fv
```
3. If the remove command doesn't complete, the most likely cause is an NFS conflict. Restart the NFS and Docker services, and then repeat the previous substep.
```
systemctl stop nfs && systemctl stop docker
systemctl start nfs && systemctl start docker
```

Dismount all file systems mounted from the master host. This step ensures that no stale mounts remain.

Identify file systems mounted from the master host. If the command returns no result, this procedure is complete.
```
awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
```

Force the file systems to dismount.

for FS in $(awk '/serviced/ { print $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs')
do
  umount -f $FS
done

If file systems remain mounted, perform a lazy dismount and restart NFS.

for FS in $(awk '/serviced/ { print $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs')
do
  umount -f -l $FS
done
systemctl restart nfs

Updating the serviced binary on a delegate host

Follow Updating the serviced binary for release 2.0.2. The procedure is identical on delegate hosts.

Logging in to the Docker registry on a delegate host

Follow Logging in to the Docker registry on each delegate host, using the registry endpoint rather than localhost. Don't skip this step. A delegate that isn't logged in can't download images, and the failure doesn't identify itself as an authentication problem.

Delegate hosts don't need their own copy of the registry password file. The file is required only on the master host, where the registry runs.

Updating the ZooKeeper image on ensemble nodes

Use this procedure to install the Docker image for ZooKeeper on ZooKeeper ensemble nodes.

Log in to the master host as root or as a user with superuser privileges.
Identify the hosts in the ZooKeeper ensemble. The result is a list of 3 or 5 hosts, separated by commas. The master host is always a node in the ensemble.
```
grep -E '^[[:space:]]*SERVICED_ZK=' /etc/default/serviced
```
Log in to a ZooKeeper ensemble node as root or as a user with superuser privileges.
Change directory to /root.
```
cd /root
```

Extract the ZooKeeper image.

yes | ./install-zenoss-isvcs-zookeeper_v*.run

Repeat the preceding four steps on each delegate host that's a node in the ZooKeeper ensemble.

Start Control Center

Use the start procedure that matches your deployment. For a multi-host deployment, follow Starting Control Center (multi-host deployment) in the 1.11.0 update instructions; the procedure is unchanged for release 2.0.2.

Starting Control Center (single-host deployment)

Log in to the master host as root or as a user with superuser privileges.

Enable serviced, if necessary.

systemctl is-enabled serviced || systemctl enable serviced

Start the Control Center service.
```
systemctl start serviced
```
Check the service status.
```
systemctl status serviced
```
Optional: Monitor the startup.
```
journalctl -u serviced -f -o cat
```

The internal services need time to start. OpenTSDB typically takes 2 to 5 minutes. A full deployment on a busy host can take 10 to 15 minutes.

Verifying the update

Use this procedure to confirm that the registry, the internal services, and the metrics pipeline are working after the update.

Confirm that the master host is logged in to the registry. The result is Login Succeeded. Replace <registry-user> with the user name you set in Configuring Docker registry authentication on the master host.
```
docker login -u <registry-user> localhost:5000
```
Confirm that the registry requires and accepts authentication.
```
curl -u <registry-user> http://127.0.0.1:5000/v2/
```
Confirm that the internal services containers are running.
```
docker ps | grep serviced-isvcs
```
Confirm that OpenTSDB responds. Replace <opentsdb-user> with the value of SERVICED_ISVCS_OPENTSDB_USERNAME.
```
curl -m 10 -u <opentsdb-user> http://localhost:4242/api/version
```
Confirm that HBase isn't restarting repeatedly. No recent output means HBase is stable.
```
docker logs --tail 20 serviced-isvcs_opentsdb 2>&1 | grep "exited: hbase"
```
In the browser interface, select a running service, and then select Internal Metrics. Graphs can remain empty for 5 to 10 minutes while Control Center collects new data. Refresh the page after that.
Confirm that per-service memory statistics report real values rather than 0B.
```
serviced service status
```
On each delegate host, confirm that images download. If an image download fails without a stated reason, the most likely cause is that the host isn't logged in to the registry. See Logging in to the Docker registry.

Note

Step 7 checks the fix for CC-4626. If the RAM columns still report 0B after the internal services have fully started, contact Support.

Post-update procedures

Removing unused images

Use this procedure to identify and remove unused Control Center images.

Log in to the master host as root or as a user with superuser privileges.
Identify the images associated with the installed version of serviced. The result is similar to IsvcsImages: [zenoss/serviced-isvcs:v76 zenoss/isvcs-zookeeper:v16].
```
serviced version | grep Images
```
Display the serviced images in the local repository.
```
docker images | awk '/REPO|isvcs/'
```
Remove unused images. Replace Image-ID with the image ID of an image for a previous version, and repeat for each unused image.
```
docker rmi Image-ID
```

Note

Keep the v76 image. Release 2.0.2 pins zenoss/serviced-isvcs:v76, which is an immutable tag. Earlier 2.0.x builds pinned a mutable development tag, so an image tagged v75-dev on this host isn't a reliable fallback.