Skip to content

Updating Control Center to release 1.10.2

This section includes procedures for updating Control Center 1.9.x to release 1.10.2.

The following list outlines recommended best practices for updating Control Center deployments:

  1. Review the release notes for this release and relevant prior releases. The latest information is provided there.
  2. On delegate hosts, most of the update steps are identical. Use screen, tmux or a similar program to establish sessions on each delegate host and perform the steps at the same time.
  3. Review and verify the settings in delegate host configuration files (/etc/default/serviced) before starting the update. Ideally, the settings on all delegate hosts are identical, except on ZooKeeper nodes and delegate hosts that do not mount the DFS.
  4. Review the update procedures before performing them. Every effort is made to avoid mistakes and anticipate needs; nevertheless, the instructions may be incorrect or inadequate for some requirements or environments.

Updating 1.9.x to 1.10.2

  1. Download the required files
  2. Stage Docker image files:
  3. Update the master host:
    1. Install the repository mirror
    2. Stop Control Center
    3. Update Docker
    4. Load image files
    5. Update the serviced binary
    6. Set internal services OpenTSDB credentials
    7. Optional: Install a security certificate
  4. Update the delegate hosts:
    1. Install the repository mirror
    2. Stop Control Center
    3. Update Docker
    4. Update the serviced binary
    5. Update the ZooKeeper image on ensemble nodes
    6. Optional: Install a security certificate
  5. Start Control Center:
  6. Perform post-upgrade procedures:

Downloading Control Center files for release 1.10.2

Use this procedure to download required files to a workstation, and then copy the files to the hosts that need them.

To perform this procedure, you need:

  • A workstation with internet access.
  • Permission to download files from delivery.zenoss.io. Customers can request permission by filing a ticket at the Zenoss Support site.
  • A secure network copy program.

Follow these steps:

  1. In a web browser, navigate to delivery.zenoss.io, and then log in.

  2. Download the self-installing Docker image files.

    • install-zenoss-serviced-isvcs:v73.run
    • install-zenoss-isvcs-zookeeper:v16.run

  3. Download the Control Center RPM file.

    • serviced-1.10.2-1.x86_64.rpm

  4. Identify the operating system release on Control Center hosts. Enter the following command on each Control Center host in your deployment, if necessary. All Control Center hosts should be running the same operating system release and kernel.

    cat /etc/redhat-release

  5. Download the RHEL/CentOS repository mirror file for your deployment.

    The download site provides repository mirror files containing the packages that Control Center requires. For RHEL/CentOS 7.7, use the 7.6 file. For RHEL 8.x, use the 7.9 file. - yum-mirror-centos7.2-1511-serviced-1.10.2.x86_64.rpm

    • yum-mirror-centos7.3-1611-serviced-1.10.2.x86_64.rpm

    • yum-mirror-centos7.4-1708-serviced-1.10.2.x86_64.rpm

    • yum-mirror-centos7.5-1804-serviced-1.10.2.x86_64.rpm

    • yum-mirror-centos7.6-1810-serviced-1.10.2.x86_64.rpm

    • yum-mirror-centos7.8.2003-serviced-1.10.2.x86_64.rpm

    • yum-mirror-centos7.9.2009-serviced-1.10.2.x86_64.rpm

  6. Optional: Download the Zenoss Pretty Good Privacy (PGP) key, if desired.

    You can use the Zenoss PGP key to verify Zenoss RPM files and the yum metadata of the repository mirror. 1. Download the key.

    ```sh
curl --location -o /tmp/tmp.html 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xED0A5FD2AA5A1AD7'
```

    1. Determine whether the download succeeded.

      grep -Ec '^\-\-\-\-\-BEGIN PGP' /tmp/tmp.html
      • If the result is 0, return to the previous substep.
      • If the result is 1, proceed to the next substep.

    2. Extract the key.

      awk '/^-----BEGIN PGP.*$/,/^-----END PGP.*$/' /tmp/tmp.html > ./RPM-PGP-KEY-Zenoss

  7. Use a secure copy program to copy the files to Control Center hosts.

    • Copy all files to the master host.

    • Copy the following files to to all delegate hosts:

      • RHEL/CentOS RPM file
      • Control Center RPM file
      • Zenoss PGP key file

    • Copy the Docker image file for ZooKeeper to delegate hosts that are ZooKeeper ensemble nodes.

Staging Docker image files on the master host

Before performing this procedure, verify that approximately 640MB of temporary space is available on the file system where /root is located.

Use this procedure to copy Docker image files to the Control Center master host. The files are used when Docker is fully configured.

  1. Log in to the master host as root or as a user with superuser privileges.

  2. Copy or move the archive files to /root.

  3. Add execute permission to the files.

    chmod +x /root/*.run

Staging a Docker image file on ZooKeeper ensemble nodes

Before performing this procedure, verify that approximately 170MB of temporary space is available on the file system where /root is located.

Use this procedure to add a Docker image file to the Control Center delegate hosts that are ZooKeeper ensemble nodes. Delegate hosts that are not ZooKeeper ensemble nodes do not need the file.

  1. Log in to a delegate host as root or as a user with superuser privileges.

  2. Copy or move the install-zenoss-isvcs-zookeeper-v*.run file to /root.

  3. Add execute permission to the file.

    chmod +x /root/*.run

Update the master host

The following procedures are to be performed on the master host of a multi-host deployment or on the single host in a single-host deployment.

The procedures for updating a delegate host can be found below.

Installing the repository mirror

Use this procedure to install the Zenoss repository mirror on a Control Center host. The mirror contains packages that are required on all Control Center hosts. Repeat this procedure on each host in your deployment.

  1. Log in to the target host as root or as a user with superuser privileges.

  2. Move the RPM files and the Zenoss PGP key file to /tmp.

  3. Upgrades only: Remove the existing repository mirror, if necessary.

    1. Search for the mirror.

      yum list --disablerepo=* | awk '/^yum-mirror/ { print $1}'

    2. Remove the mirror.

      Replace Old-Mirror with the name of the Zenoss repository mirror returned in the previous substep:

      yum remove Old-Mirror

  4. Install the repository mirror.

    yum install /tmp/yum-mirror-*.rpm

    The yum command copies the contents of the RPM file to /opt/zenoss-repo-mirror.

  5. Optional: Install the Zenoss PGP key, and then test the package files, if desired.

    1. Move the Zenoss PGP key to the mirror directory.

      mv /tmp/RPM-PGP-KEY-Zenoss /opt/zenoss-repo-mirror

    2. Install the key.

      rpm --import /opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss

    3. Test the repository mirror package file.

      rpm -K /tmp/yum-mirror-*.rpm

      On success, the result includes the file name and the following information:

      (sha1) dsa sha1 md5 gpg OK

    4. Test the Control Center package file.

      rpm -K /tmp/serviced-VERSION-1.x86_64.rpm

  6. Optional: Update the configuration file of the Zenoss repository mirror to enable PGP key verification, if desired.

    1. Open the repository mirror configuration file (/etc/yum.repos.d/zenoss-mirror.repo) with a text editor, and then add the following lines to the end of the file.

      repo_gpgcheck=1
gpgkey=file:///opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss

    2. Save the file, and then close the editor.

    3. Update the yum metadata cache.

      yum makecache fast

      The cache update process includes the following prompt:

      Retrieving key from file:///opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
Importing GPG key 0xAA5A1AD7:
 Userid     : "Zenoss, Inc. <dev@zenoss.com>"
 Fingerprint: f31f fd84 6a23 b3d5 981d a728 ed0a 5fd2 aa5a 1ad7
 From       : /opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
Is this ok [y/N]:

      Enter y.

  7. Move the Control Center package file to the mirror directory.

    mv /tmp/serviced-VERSION-1.x86_64.rpm /opt/zenoss-repo-mirror

  8. Optional: Delete the mirror package file, if desired.

    rm /tmp/yum-mirror-*.rpm

Stopping Control Center on the master host

Use this procedure to stop the Control Center service (serviced) on the master host.

  1. Log in to the master host as root or as a user with superuser privileges.

  2. Stop the top-level service serviced is managing, if necessary.

    1. Show the status of running services.

      serviced service status

      The top-level service is the service listed immediately below the headings line. - If the status of the top-level service and all child services is stopped, proceed to the next step. - If the status of the top-level service and all child services is not stopped, perform the remaining substeps.

    2. Stop the top-level service.

      serviced service stop Zenoss.resmgr

    3. Monitor the stop.

      serviced service status

      When the status of the top-level service and all child services is stopped, proceed to the next step.

  3. Stop the Control Center service.

    systemctl stop serviced

  4. Ensure that no containers remain in the local repository.

    1. Display the identifiers of all containers, running and exited.

      docker ps -qa
      • If the command returns no result, stop. This procedure is complete.
      • If the command returns a result, perform the following substeps.

    2. Remove all remaining containers.

      docker ps -qa | xargs --no-run-if-empty docker rm -fv

    3. Display the identifiers of all containers, running and exited.

      docker ps -qa
      • If the command returns no result, stop. This procedure is complete.
      • If the command returns a result, perform the remaining substeps.

    4. Disable the automatic startup of serviced.

      systemctl disable serviced

    5. Reboot the host.

      reboot

    6. Log in to the master host as root, or as a user with superuser privileges.

    7. Enable the automatic startup of serviced.

      systemctl enable serviced

Updating Docker 19.03.11 to 20.10.17

Use this procedure to update Docker to version 20.10.17.

  1. Log in as root or as a user with superuser privileges.

  2. Determine which version of Docker is installed.

    docker info 2>/dev/null | awk '/^ Server/ { print $3 }'
    • If the result is 20.10.17, stop. This procedure is unnecessary; proceed to the next one.
    • If the result is 19.03.11, perform the remaining steps in this procedure.
    • If the result is any earlier version, proceed to Updating Control Center to release 1.10.2.

  3. Update the operating system, if necessary.

    1. Determine which release is installed.

      cat /etc/redhat-release
      • If the result is greater than 7.2, proceed to the next step.
      • If the result is less than or equal to 7.1, perform the remaining substeps.

    2. Disable automatic start of serviced.

      systemctl disable serviced

    3. Update the operating system, and then restart the host.

      The following commands require internet access or a local mirror of operating system packages.

      yum makecache fast && yum update --skip-broken && reboot

    4. Log in as root or as a user with superuser privileges.

    5. Enable automatic start of serviced.

      systemctl enable serviced

  4. Update the Linux kernel, if necessary.

    1. Determine which kernel version is installed.

      uname -r

      If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.

    2. Disable automatic start of serviced.

      systemctl disable serviced

    3. Update the kernel, and then restart the host.

      The following commands require internet access or a local mirror of operating system packages.

      yum makecache fast && yum update kernel --skip-broken && reboot

    4. Log in as root, or as a user with superuser privileges.

    5. Enable automatic start of serviced.

      systemctl enable serviced

  5. Stop the Docker service.

    systemctl stop docker

  6. Remove Docker 19.03.11.

    1. Remove without checking dependencies.

      rpm -e --nodeps docker-ce

      The result may include the following message, which can be ignored:

      /usr/bin/dockerd has not been configured as an alternative for dockerd

    2. Clean the yum databases.

      yum clean all && yum makecache fast

  7. Install Docker CE 20.10.17.

    yum install --enablerepo=zenoss-mirror docker-ce-20.10.17-3.el7 docker-ce-cli

    If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.

  8. Start the Docker service.

    systemctl start docker

Loading image files

Use this procedure to load images into the local Docker registry on a host.

  1. Log in to the host as root or as a user with superuser privileges.

  2. Change directory to /root.

    cd /root

  3. Load the images.

    for image in install-zenoss-*.run
do
  /bin/echo -en "\nLoading $image..."
  yes | ./$image
done

  4. List the images in the registry.

    docker images

    The result should show one image for each archive file.

  5. Optional: Delete the archive files, if desired.

    rm -i ./install-zenoss-*.run

Updating the serviced binary for release 1.10.2

Use this procedure to update the serviced binary on a Control Center host. Perform this procedure on each host in your Control Center deployment.

In multi-host deployments, stop serviced on the master host first.

  1. Log in to the host as root or as a user with superuser privileges.

  2. Start the docker service, if necessary.

    systemctl is-active docker || systemctl start docker

  3. Disable SELinux temporarily, if necessary.

    1. Determine the current mode.

      sestatus | awk '/Current mode:/ { print $3 }'
      • If the result is not enforcing, proceed to step 4.
      • If the result is enforcing, perform the next substep.

    2. Disable SELinux temporarily.

      setenforce 0

  4. Save the current serviced configuration file as a reference and set permissions to read-only.

    mv /etc/default/serviced /etc/default/serviced-pre-1.10.2
chmod 0440 /etc/default/serviced-pre-1.10.2

  5. Create a temporary directory, if necessary. The RPM installation process writes a script to TMP and starts it with exec to migrate Elasticsearch.

    1. Determine whether the /tmp directory is mounted on a partition with the noexec option set.

      awk '$2 ~ /\/tmp/ { if ($4 ~ /noexec/) print "protected" }' < /proc/mounts

      If the command returns a result, perform the following substeps. Otherwise, proceed to step 5.

    2. Create a temporary storage variable for TMP, if necessary.

      test -n "$TMP" && tempTMP=$TMP

    3. Create a temporary directory.

      mkdir $HOME/tempdir && export TMP=$HOME/tempdir

  6. Install the new serviced RPM package.

    yum install --enablerepo=zenoss-mirror /opt/zenoss-repo-mirror/serviced-1.10.2-1.x86_64.rpm

    If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.

  7. Enable SELinux, if necessary. Perform this step only if you disabled SELinux in step 3:

    setenforce 1

  8. Restore the previous value of TMP, if necessary.

    test -n "$tempTMP" && export TMP=$tempTMP

  9. Make a backup copy of the new configuration file and set permissions to read-only.

    cp /etc/default/serviced /etc/default/serviced-1.10.2-orig
chmod 0440 /etc/default/serviced-1.10.2-orig

  10. Compare the new configuration file with the configuration file of the previous release.

    1. Identify the configuration files to compare.

      ls -l /etc/default/serviced*

      The original versions of the configuration files should end with orig, but you may have to compare the dates of the files.

    2. Compare the new and previous configuration files.Replace New-Version with the name of the new configuration file, and replace Previous-Version with the name of the previous configuration file:

      diff New-Version Previous-Version

      For example, to compare versions 1.9.1 and the most recent version, enter the following command:

      diff /etc/default/serviced-1.9.1-orig /etc/default/serviced-1.10.2-orig

    3. If the command returns no result, restore the backup of the previous configuration file.

      cp /etc/default/serviced-pre-1.10.2 /etc/default/serviced && chmod 0644 /etc/default/serviced

    4. If the command returns a result, restore the backup of the previous configuration file, and then optionally, use the results to edit the restored version.

      cp /etc/default/serviced-pre-1.10.2 /etc/default/serviced && chmod 0644 /etc/default/serviced

      For more information about configuring a host, see Configuration variables.

  11. Reload the systemd manager configuration.

    systemctl daemon-reload

Setting the internal services OpenTSDB credentials

Starting with release 1.10.2, Control Center requires a username and password to run the internal services OpenTSDB database.

  1. Log in to the master host as root or as a user with superuser privileges.

  2. Edit the Control Center configuration file.

    1. Open /etc/default/serviced in a text editor.
    2. Locate the line for the SERVICED_ISVCS_OPENTSDB_USERNAME variable, and then make a copy of the line, immediately below the original.
    3. Remove the number sign character (#) from the beginning of the line.
    4. Enter a username for the OpenTSDB server. You can choose any username you wish.
    5. Locate the line for the SERVICED_ISVCS_OPENTSDB_PASSWORD variable, and then make a copy of the line, immediately below the original.
    6. Remove the number sign character (#) from the beginning of the line.
    7. Enter a password for the OpenTSDB server. You can choose any password you wish.
    8. Save the file, and then close the editor.

  3. Verify the credentials in the serviced configuration file.

    grep -E '^[[:space:]]*SERVICED_ISVCS_OPENTSDB' /etc/default/serviced

  4. Restart serviced.

    systemctl restart serviced

Optional: Installing a security certificate

The default, insecure certificate that Control Center uses for TLS-encrypted communications are based on a public certificate compiled into serviced. Use this procedure to replace the default certificate files with your own files.

  • If you are using virtual host public endpoints for your Zenoss Service Dynamics deployment, you need a wildcard certificate or a subject alternative name (SAN) certificate.
  • If your end users access the browser interface through a reverse proxy, the reverse proxy may provide the browser with its own SSL certificate. If so, please contact Zenoss Support for additional assistance.

To perform this procedure, you need valid certificate files. For information about generating a self-signed certificate, see Creating a self-signed security certificate.

To use your own certificate files, perform this procedure on the Control Center master host and on each Control Center delegate host in your environment.

Follow these steps:

  1. Log in to the host as root or as a user with superuser privileges.

  2. Use a secure copy program to copy the key and certificate files to /tmp.

  3. Move the key file to the /etc/pki/tls/private directory. Replace <KEY_FILE> with the name of your key file:

    mv /tmp/<KEY_FILE>.key /etc/pki/tls/private

  4. Move the certificate file to the /etc/pki/tls/certs directory. Replace <CERT_FILE> with the name of your certificate file:

    mv /tmp/<CERT_FILE>.crt /etc/pki/tls/certs

  5. Updates only: Create a backup copy of the Control Center configuration file. Do not perform this step for a fresh install:

    cp /etc/default/serviced /etc/default/serviced.before-cert-files

  6. Edit the Control Center configuration file.

    1. Open /etc/default/serviced in a text editor.
    2. Locate the line for the SERVICED_KEY_FILE variable, and then make a copy of the line, immediately below the original.
    3. Remove the number sign character (#) from the beginning of the line.
    4. Replace the default value with the full pathname of your key file.
    5. Locate the line for the SERVICED_CERT_FILE variable, and then make a copy of the line, immediately below the original.
    6. Remove the number sign character (#) from the beginning of the line.
    7. Replace the default value with the full pathname of your certificate file.
    8. Save the file, and then close the editor.

  7. Verify the settings in the configuration file.

    grep -E '^[[:space:]]*[A-Z_]+' /etc/default/serviced

  8. Updates only: Reload the systemd manager configuration. Do not perform this step for a fresh install:

    systemctl daemon-reload

Update the delegate hosts

Installing the repository mirror

Use this procedure to install the Zenoss repository mirror on a Control Center host. The mirror contains packages that are required on all Control Center hosts. Repeat this procedure on each host in your deployment.

  1. Log in to the target host as root or as a user with superuser privileges.

  2. Move the RPM files and the Zenoss PGP key file to /tmp.

  3. Upgrades only: Remove the existing repository mirror, if necessary.

    1. Search for the mirror.

      yum list --disablerepo=* | awk '/^yum-mirror/ { print $1}'

    2. Remove the mirror.

      Replace Old-Mirror with the name of the Zenoss repository mirror returned in the previous substep:

      yum remove Old-Mirror

  4. Install the repository mirror.

    yum install /tmp/yum-mirror-*.rpm

    The yum command copies the contents of the RPM file to /opt/zenoss-repo-mirror.

  5. Optional: Install the Zenoss PGP key, and then test the package files, if desired.

    1. Move the Zenoss PGP key to the mirror directory.

      mv /tmp/RPM-PGP-KEY-Zenoss /opt/zenoss-repo-mirror

    2. Install the key.

      rpm --import /opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss

    3. Test the repository mirror package file.

      rpm -K /tmp/yum-mirror-*.rpm

      On success, the result includes the file name and the following information:

      (sha1) dsa sha1 md5 gpg OK

    4. Test the Control Center package file.

      rpm -K /tmp/serviced-VERSION-1.x86_64.rpm

  6. Optional: Update the configuration file of the Zenoss repository mirror to enable PGP key verification, if desired.

    1. Open the repository mirror configuration file (/etc/yum.repos.d/zenoss-mirror.repo) with a text editor, and then add the following lines to the end of the file.

      repo_gpgcheck=1
gpgkey=file:///opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss

    2. Save the file, and then close the editor.

    3. Update the yum metadata cache.

      yum makecache fast

      The cache update process includes the following prompt:

      Retrieving key from file:///opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
Importing GPG key 0xAA5A1AD7:
 Userid     : "Zenoss, Inc. <dev@zenoss.com>"
 Fingerprint: f31f fd84 6a23 b3d5 981d a728 ed0a 5fd2 aa5a 1ad7
 From       : /opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
Is this ok [y/N]:

      Enter y.

  7. Move the Control Center package file to the mirror directory.

    mv /tmp/serviced-VERSION-1.x86_64.rpm /opt/zenoss-repo-mirror

  8. Optional: Delete the mirror package file, if desired.

    rm /tmp/yum-mirror-*.rpm

Stopping Control Center on a delegate host

Use this procedure to stop the Control Center service (serviced) on a delegate host in a multi-host deployment. Repeat this procedure on each delegate host in your deployment.

Before performing this procedure on any delegate host, stop Control Center on the master host.

  1. Log in to the delegate host as root or as a user with superuser privileges.

  2. Stop the Control Center service.

    systemctl stop serviced

  3. Ensure that no containers remain in the local repository.

    1. Display the identifiers of all containers, running and exited.

      docker ps -qa
      • If the command returns no result, proceed to the next step.
      • If the command returns a result, perform the following substeps.

    2. Remove all remaining containers.

      docker ps -qa | xargs --no-run-if-empty docker rm -fv
      • If the remove command completes, proceed to the next step.
      • If the remove command does not complete, the most likely cause is an NFS conflict. Perform the following substeps.

    3. Stop the NFS and Docker services.

      systemctl stop nfs && systemctl stop docker

    4. Start the NFS and Docker services.

      systemctl start nfs && systemctl start docker

    5. Repeat the attempt to remove all remaining containers.

      docker ps -qa | xargs --no-run-if-empty docker rm -fv
      • If the remove command completes, proceed to the next step.
      • If the remove command does not complete, perform the remaining substeps.

    6. Disable the automatic startup of serviced.

      systemctl disable serviced

    7. Reboot the host.

      reboot

    8. Log in to the delegate host as root, or as a user with superuser privileges.

    9. Enable the automatic startup of serviced.

      systemctl enable serviced

  4. Dismount all filesystems mounted from the Control Center master host.

    This step ensures no stale mounts remain when the storage on the master host is replaced.

    1. Identify filesystems mounted from the master host.

      awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
      • If the preceding command returns no result, stop. This procedure is complete.
      • If the preceding command returns a result, perform the following substeps.

    2. Force the filesystems to dismount.

      for FS in $(awk '/serviced/ { print $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs')
do
  umount -f $FS
done

    3. Identify filesystems mounted from the master host.

      awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
      • If the preceding command returns no result, stop. This procedure is complete.
      • If the preceding command returns a result, perform the following substeps.

    4. Perform a lazy dismount.

      for FS in $(awk '/serviced/ { print $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs')
do
  umount -f -l $FS
done

    5. Restart the NFS service.

      systemctl restart nfs

    6. Determine whether any filesystems remain mounted from the master host.

      awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
      • If the preceding command returns no result, stop. This procedure is complete.
      • If the preceding command returns a result, perform the remaining substeps.

    7. Disable the automatic startup of serviced.

      systemctl disable serviced

    8. Reboot the host.

      reboot

    9. Log in to the delegate host as root, or as a user with superuser privileges.

    10. Enable the automatic startup of serviced.

      systemctl enable serviced

Updating Docker 19.03.11 to 20.10.17

Use this procedure to update Docker to version 20.10.17.

  1. Log in as root or as a user with superuser privileges.

  2. Determine which version of Docker is installed.

    docker info 2>/dev/null | awk '/^ Server/ { print $3 }'
    • If the result is 20.10.17, stop. This procedure is unnecessary; proceed to the next one.
    • If the result is 19.03.11, perform the remaining steps in this procedure.
    • If the result is any earlier version, proceed to Updating Control Center to release 1.10.2.

  3. Update the operating system, if necessary.

    1. Determine which release is installed.

      cat /etc/redhat-release
      • If the result is greater than 7.2, proceed to the next step.
      • If the result is less than or equal to 7.1, perform the remaining substeps.

    2. Disable automatic start of serviced.

      systemctl disable serviced

    3. Update the operating system, and then restart the host.

      The following commands require internet access or a local mirror of operating system packages.

      yum makecache fast && yum update --skip-broken && reboot

    4. Log in as root or as a user with superuser privileges.

    5. Enable automatic start of serviced.

      systemctl enable serviced

  4. Update the Linux kernel, if necessary.

    1. Determine which kernel version is installed.

      uname -r

      If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.

    2. Disable automatic start of serviced.

      systemctl disable serviced

    3. Update the kernel, and then restart the host.

      The following commands require internet access or a local mirror of operating system packages.

      yum makecache fast && yum update kernel --skip-broken && reboot

    4. Log in as root, or as a user with superuser privileges.

    5. Enable automatic start of serviced.

      systemctl enable serviced

  5. Stop the Docker service.

    systemctl stop docker

  6. Remove Docker 19.03.11.

    1. Remove without checking dependencies.

      rpm -e --nodeps docker-ce

      The result may include the following message, which can be ignored:

      /usr/bin/dockerd has not been configured as an alternative for dockerd

    2. Clean the yum databases.

      yum clean all && yum makecache fast

  7. Install Docker CE 20.10.17.

    yum install --enablerepo=zenoss-mirror docker-ce-20.10.17-3.el7 docker-ce-cli

    If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.

  8. Start the Docker service.

    systemctl start docker

Updating the serviced binary for release 1.10.2

Use this procedure to update the serviced binary on a Control Center host. Perform this procedure on each host in your Control Center deployment.

In multi-host deployments, stop serviced on the master host first.

  1. Log in to the host as root or as a user with superuser privileges.

  2. Start the docker service, if necessary.

    systemctl is-active docker || systemctl start docker

  3. Disable SELinux temporarily, if necessary.

    1. Determine the current mode.

      sestatus | awk '/Current mode:/ { print $3 }'
      • If the result is not enforcing, proceed to step 4.
      • If the result is enforcing, perform the next substep.

    2. Disable SELinux temporarily.

      setenforce 0

  4. Save the current serviced configuration file as a reference and set permissions to read-only.

    mv /etc/default/serviced /etc/default/serviced-pre-1.10.2
chmod 0440 /etc/default/serviced-pre-1.10.2

  5. Create a temporary directory, if necessary. The RPM installation process writes a script to TMP and starts it with exec to migrate Elasticsearch.

    1. Determine whether the /tmp directory is mounted on a partition with the noexec option set.

      awk '$2 ~ /\/tmp/ { if ($4 ~ /noexec/) print "protected" }' < /proc/mounts

      If the command returns a result, perform the following substeps. Otherwise, proceed to step 5.

    2. Create a temporary storage variable for TMP, if necessary.

      test -n "$TMP" && tempTMP=$TMP

    3. Create a temporary directory.

      mkdir $HOME/tempdir && export TMP=$HOME/tempdir

  6. Install the new serviced RPM package.

    yum install --enablerepo=zenoss-mirror /opt/zenoss-repo-mirror/serviced-1.10.2-1.x86_64.rpm

    If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.

  7. Enable SELinux, if necessary. Perform this step only if you disabled SELinux in step 3:

    setenforce 1

  8. Restore the previous value of TMP, if necessary.

    test -n "$tempTMP" && export TMP=$tempTMP

  9. Make a backup copy of the new configuration file and set permissions to read-only.

    cp /etc/default/serviced /etc/default/serviced-1.10.2-orig
chmod 0440 /etc/default/serviced-1.10.2-orig

  10. Compare the new configuration file with the configuration file of the previous release.

    1. Identify the configuration files to compare.

      ls -l /etc/default/serviced*

      The original versions of the configuration files should end with orig, but you may have to compare the dates of the files.

    2. Compare the new and previous configuration files.Replace New-Version with the name of the new configuration file, and replace Previous-Version with the name of the previous configuration file:

      diff New-Version Previous-Version

      For example, to compare versions 1.9.1 and the most recent version, enter the following command:

      diff /etc/default/serviced-1.9.1-orig /etc/default/serviced-1.10.2-orig

    3. If the command returns no result, restore the backup of the previous configuration file.

      cp /etc/default/serviced-pre-1.10.2 /etc/default/serviced && chmod 0644 /etc/default/serviced

    4. If the command returns a result, restore the backup of the previous configuration file, and then optionally, use the results to edit the restored version.

      cp /etc/default/serviced-pre-1.10.2 /etc/default/serviced && chmod 0644 /etc/default/serviced

      For more information about configuring a host, see Configuration variables.

  11. Reload the systemd manager configuration.

    systemctl daemon-reload

Updating the ZooKeeper image on ensemble nodes

Use this procedure to install a new Docker image for ZooKeeper on ZooKeeper ensemble nodes.

  1. Log in to the master host as root or as a user with superuser privileges.

  2. Identify the hosts in the ZooKeeper ensemble.

    grep -E '^[[:space:]]*SERVICED_ZK=' /etc/default/serviced

    The result is a list of 3 or 5 hosts, separated by the comma character (,). The master host is always a node in the ZooKeeper ensemble.

  3. Log in to a ZooKeeper ensemble node as root, or as a user with superuser privileges.

  4. Change directory to /root.

    cd /root

  5. Extract the ZooKeeper image.

    yes | ./install-zenoss-isvcs-zookeeper_v*.run

  6. Optional: Delete the archive file.

    rm -i ./install-zenoss-isvcs-zookeeper_v*.run

  7. Repeat the preceding four steps on each delegate that is a node in the ZooKeeper ensemble.

Optional: Installing a security certificate

The default, insecure certificate that Control Center uses for TLS-encrypted communications are based on a public certificate compiled into serviced. Use this procedure to replace the default certificate files with your own files.

  • If you are using virtual host public endpoints for your Zenoss Service Dynamics deployment, you need a wildcard certificate or a subject alternative name (SAN) certificate.
  • If your end users access the browser interface through a reverse proxy, the reverse proxy may provide the browser with its own SSL certificate. If so, please contact Zenoss Support for additional assistance.

To perform this procedure, you need valid certificate files. For information about generating a self-signed certificate, see Creating a self-signed security certificate.

To use your own certificate files, perform this procedure on the Control Center master host and on each Control Center delegate host in your environment.

Follow these steps:

  1. Log in to the host as root or as a user with superuser privileges.

  2. Use a secure copy program to copy the key and certificate files to /tmp.

  3. Move the key file to the /etc/pki/tls/private directory. Replace <KEY_FILE> with the name of your key file:

    mv /tmp/<KEY_FILE>.key /etc/pki/tls/private

  4. Move the certificate file to the /etc/pki/tls/certs directory. Replace <CERT_FILE> with the name of your certificate file:

    mv /tmp/<CERT_FILE>.crt /etc/pki/tls/certs

  5. Updates only: Create a backup copy of the Control Center configuration file. Do not perform this step for a fresh install:

    cp /etc/default/serviced /etc/default/serviced.before-cert-files

  6. Edit the Control Center configuration file.

    1. Open /etc/default/serviced in a text editor.
    2. Locate the line for the SERVICED_KEY_FILE variable, and then make a copy of the line, immediately below the original.
    3. Remove the number sign character (#) from the beginning of the line.
    4. Replace the default value with the full pathname of your key file.
    5. Locate the line for the SERVICED_CERT_FILE variable, and then make a copy of the line, immediately below the original.
    6. Remove the number sign character (#) from the beginning of the line.
    7. Replace the default value with the full pathname of your certificate file.
    8. Save the file, and then close the editor.

  7. Verify the settings in the configuration file.

    grep -E '^[[:space:]]*[A-Z_]+' /etc/default/serviced

  8. Updates only: Reload the systemd manager configuration. Do not perform this step for a fresh install:

    systemctl daemon-reload

Start Control Center

There are separate start-up procedures for single- and multi-host deployments of Control Center. Please use the procedure appropriate to your deployment.

Starting Control Center (single-host deployment)

Use this procedure to start Control Center in a single-host deployment. The default configuration of the Control Center service (serviced) is to start when the host starts. This procedure is only needed after stopping serviced to perform maintenance tasks.

  1. Log in to the master host as root or as a user with superuser privileges.

  2. Enable serviced, if necessary.

    systemctl is-enabled serviced || systemctl enable serviced

  3. Start the Control Center service.

    systemctl start serviced

  4. Optional: Monitor the startup, if desired.

    journalctl -u serviced -f -o cat

Starting Control Center (multi-host deployment)

Use this procedure to start Control Center in a multi-host deployment. The default configuration of the Control Center service (serviced) is to start when the host starts. This procedure is only needed after stopping serviced to perform maintenance tasks.

  1. Log in to the master host as root or as a user with superuser privileges.

  2. Enable serviced, if necessary.

    systemctl is-enabled serviced || systemctl enable serviced

  3. Identify the hosts in the ZooKeeper ensemble.

    grep -E '^[[:space:]]*SERVICED_ZK=' /etc/default/serviced

    The result is a list of 1, 3, or 5 hosts, separated by the comma character (,). The master host is always a node in the ZooKeeper ensemble.

  4. In separate windows, log in to each of the delegate hosts that are nodes in the ZooKeeper ensemble as root, or as a user with superuser privileges.

  5. On all ensemble hosts, start serviced.

    The window of time for starting a ZooKeeper ensemble is relatively short. The goal of this step is to start Control Center on each ensemble node at about the same time, so that each node can participate in electing the leader.

    systemctl start serviced

  6. On the master host, check the status of the ZooKeeper ensemble.

    1. Attach to the container of the ZooKeeper service.

      docker exec -it serviced-isvcs_zookeeper bash

    2. Query the master host and identify its role in the ensemble.

      Replace Master with the hostname or IP address of the master host:

      { echo stats; sleep 1; } | nc Master 2181 | grep Mode

      The result includes leader or follower. When multiple hosts rely on the ZooKeeper instance on the master host, the result includes standalone.

    3. Query the other delegate hosts to identify their role in the ensemble.

      Replace Delegate with the hostname or IP address of a delegate host:

      { echo stats; sleep 1; } | nc Delegate 2181 | grep Mode

    4. Detach from the container of the ZooKeeper service.

      exit

    If none of the nodes reports that it is the ensemble leader within a few minutes of starting serviced, reboot the ensemble hosts.

  7. Log in to each of the delegate hosts that are not nodes in the ZooKeeper ensemble as root, or as a user with superuser privileges, and then start serviced.

    systemctl start serviced

  8. Optional: Monitor the startup, if desired.

    journalctl -u serviced -f -o cat

Post-upgrade procedures

Removing unused images

Use this procedure to identify and remove unused Control Center images.

  1. Log in to the master host as root or as a user with superuser privileges.

  2. Identify the images associated with the installed version of serviced.

    serviced version | grep Images

    Example result:

    IsvcsImages: [zenoss/serviced-isvcs:v73 zenoss/isvcs-zookeeper:v16]

  3. Start Docker, if necessary.

    systemctl status docker || systemctl start docker

  4. Display the serviced images in the local repository.

    docker images | awk '/REPO|isvcs/'

    Example result (edited to fit):

    REPOSITORY                     TAG       IMAGE ID
zenoss/serviced-isvcs          v73       0bd933b3fb2f
zenoss/serviced-isvcs          v70       c19f1e317158
zenoss/isvcs-zookeeper         v16       7ea8c92ca1ad
zenoss/isvcs-zookeeper         v14       0ff3b3117fb8

    The example result shows the current versions and one set of previous versions. Your result may include additional previous versions and will show different images IDs.

  5. Remove unused images.

    Replace Image-ID with the image ID of an image for a previous version.

    docker rmi Image-ID

    Repeat this command for each unused image.

Optional: Adding ZooKeeper security to an existing deployment

Use this procedure to secure Zookeeper in an existing Control Center deployment. For more information, see Zookeeper security.

  1. Set the Control Center variables for Zookeeper security.
  2. Stop Control Center on the master host and all delegate hosts.

  3. On each Control Center host, remove the Zookeeper data directory.

    1. Log in to the host as root or as a user with superuser privileges.

    2. Remove the directory.

      rm -rf --preserve-root /opt/serviced/var/iscvs/zookeeper

  4. Start Control Center.

  5. Optional: Secure Zookeeper data nodes.