Updating Control Center to release 1.7.0

This section includes procedures for updating Control Center to release 1.7.0.

The following table summarizes the new features of the update paths that are included in this section.

From To New features

1.5.x or 1.6.3

1.7.0

The following changes are addressed during this update.

This release replaces Docker Community Edition (CE) 17.09.0 with Docker CE 18.09.6.
OpenTSDB is updated to 2.3.1, so the update requires a new version of the serviced-isvcs Docker image.

1.4.1 or 1.4.2

1.7.0

The following changes are addressed during this update.

Automatic restart when a service fails 3 health checks in a row.
RHEL/CentOS 7.4, 7.5, and 7.6 are added; RHEL/CentOS 7.1 is withdrawn. The update process includes an optional step for updating the operating system.
On RHEL/CentOS 7.4 and up, there may be a file locking defect in NFS 4.1. To avoid the issue, delegate hosts are configured to use NFS 4.0.
The serviced-zenossdbpack maintenance script is moved from /etc/cron.weekly to /opt/serviced/bin and a new cron job, /etc/cron.d/cron_zenossdbpack, is installed to invoke the maintenance script. By default, the script runs every Sunday at midnight.
This release replaces Docker Community Edition (CE) 17.03.1 with Docker CE 18.09.6.

The following list outlines recommended best practices for updating Control Center deployments:

Review the release notes for this release and relevant prior releases. The latest information is provided there.
Compare the Docker images that accompany this release with the images of the installed release, and determine whether new image files need to be downloaded and installed. For more information, see Releases and image tags.
On delegate hosts, most of the update steps are identical. Use screen, tmux or a similar program to establish sessions on each delegate host and perform the steps at the same time.
Review and verify the settings in delegate host configuration files (/etc/default/serviced) before starting the update. Ideally, the settings on all delegate hosts are identical, except on ZooKeeper nodes and delegate hosts that do not mount the DFS.
Review the update procedures before performing them. Every effort is made to avoid mistakes and anticipate needs; nevertheless, the instructions may be incorrect or inadequate for some requirements or environments.

Updating from 1.5.x or 1.6.3

The following procedures cover the upgrade of Control Center (versions 1.5.x or 1.6.3) to 1.7.0.

If you are upgrading from 1.4.1 or 1.4.2, please see Updating from 1.4.1 or 1.4.2.

Outline

Download required files
Install the repository mirror
Stage Docker image files
Update the master host:
Update delegate hosts:
Start Control Center:
- Start Control Center (single-host deployment)
- Start Control Center (multi-host deployment)
Perform post-upgrade procedures:
- Set the connection timeout of a resource pool

Downloading Control Center files for release 1.7.0

Use this procedure to download required files to a workstation, and then copy the files to the hosts that need them.

To perform this procedure, you need:

A workstation with internet access.
Permission to download files from delivery.zenoss.io. Customers can request permission by filing a ticket at the Zenoss Support site.
A secure network copy program.

Follow these steps:

In a web browser, navigate to delivery.zenoss.io, and then log in.
Download the self-installing Docker image files.
- install-zenoss-serviced-isvcs:v63.run
- install-zenoss-isvcs-zookeeper:v11.run
Download the Control Center RPM file.
- serviced-1.7.0-1.x86_64.rpm
Identify the operating system release on Control Center hosts. Enter the following command on each Control Center host in your deployment, if necessary. All Control Center hosts should be running the same operating system release and kernel.
```
cat /etc/redhat-release
```
Download the RHEL/CentOS repository mirror file for your deployment. The download site provides repository mirror files containing the packages that Control Center requires. For RHEL/CentOS 7.7 and 7.8, use the 7.6 file.
- yum-mirror-centos7.2-1511-serviced-1.6.5.x86_64.rpm
- yum-mirror-centos7.3-1611-serviced-1.6.5.x86_64.rpm
- yum-mirror-centos7.4-1708-serviced-1.6.5.x86_64.rpm
- yum-mirror-centos7.5-1804-serviced-1.6.5.x86_64.rpm
- yum-mirror-centos7.6-1810-serviced-1.6.5.x86_64.rpm
  
  Currently, Red Hat Enterprise Linux (RHEL) 8.x and CentOS 8.x are unsupported.
Optional: Download the Zenoss Pretty Good Privacy (PGP) key, if desired. You can use the Zenoss PGP key to verify Zenoss RPM files and the yum metadata of the repository mirror.
1. Download the key.
```
curl --location -o /tmp/tmp.html 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xED0A5FD2AA5A1AD7'
```
2. Determine whether the download succeeded.
```
grep -Ec '^\-\-\-\-\-BEGIN PGP' /tmp/tmp.html
```
  - If the result is 0, return to the previous substep.
  - If the result is 1, proceed to the next substep.
3. Extract the key.
```
awk '/^-----BEGIN PGP.*$/,/^-----END PGP.*$/' /tmp/tmp.html > ./RPM-PGP-KEY-Zenoss
```
Use a secure copy program to copy the files to Control Center hosts.
- Copy all files to the master host.
- Copy the RHEL/CentOS RPM file, the Control Center RPM file, and the Zenoss PGP key file to all delegate hosts.
- Copy the Docker image file for ZooKeeper to delegate hosts that are ZooKeeper ensemble nodes.

Installing the repository mirror

Use this procedure to install the Zenoss repository mirror on a Control Center host. The mirror contains packages that are required on all Control Center hosts. Repeat this procedure on each host in your deployment.

Log in to the target host as root or as a user with superuser privileges.
Move the RPM files and the Zenoss PGP key file to /tmp.
Upgrades only: Remove the existing repository mirror, if necessary.
1. Search for the mirror.
```
yum list --disablerepo=* | awk '/^yum-mirror/ { print $1}'
```
2. Remove the mirror.
  
  Replace Old-Mirror with the name of the Zenoss repository mirror returned in the previous substep:
```
yum remove Old-Mirror
```
Install the repository mirror.
```
yum install /tmp/yum-mirror-*.rpm
```
The yum command copies the contents of the RPM file to /opt/zenoss-repo-mirror.
Optional: Install the Zenoss PGP key, and then test the package files, if desired.
1. Move the Zenoss PGP key to the mirror directory.
```
mv /tmp/RPM-PGP-KEY-Zenoss /opt/zenoss-repo-mirror
```
2. Install the key.
```
rpm --import /opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
```
3. Test the repository mirror package file.
```
rpm -K /tmp/yum-mirror-*.rpm
```
  On success, the result includes the file name and the following information:
```
(sha1) dsa sha1 md5 gpg OK
```
4. Test the Control Center package file.
```
rpm -K /tmp/serviced-VERSION-1.x86_64.rpm
```

Optional: Update the configuration file of the Zenoss repository mirror to enable PGP key verification, if desired.

Open the repository mirror configuration file (/etc/yum.repos.d/zenoss-mirror.repo) with a text editor, and then add the following lines to the end of the file.
```
repo_gpgcheck=1
gpgkey=file:///opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
```
Save the file, and then close the editor.

Update the yum metadata cache.

yum makecache fast

The cache update process includes the following prompt:

Retrieving key from file:///opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
Importing GPG key 0xAA5A1AD7:
 Userid     : "Zenoss, Inc. <dev@zenoss.com>"
 Fingerprint: f31f fd84 6a23 b3d5 981d a728 ed0a 5fd2 aa5a 1ad7
 From       : /opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
Is this ok [y/N]:

Enter y.

Move the Control Center package file to the mirror directory.

mv /tmp/serviced-VERSION-1.x86_64.rpm /opt/zenoss-repo-mirror

Optional: Delete the mirror package file, if desired.
```
rm /tmp/yum-mirror-*.rpm
```

Staging Docker image files on the master host

Before performing this procedure, verify that approximately 640MB of temporary space is available on the file system where /root is located.

Use this procedure to copy Docker image files to the Control Center master host. The files are used when Docker is fully configured.

Log in to the master host as root or as a user with superuser privileges.
Copy or move the archive files to /root.
Add execute permission to the files.
```
chmod +x /root/*.run
```

Staging a Docker image file on ZooKeeper ensemble nodes

Before performing this procedure, verify that approximately 170MB of temporary space is available on the file system where /root is located.

Use this procedure to add a Docker image file to the Control Center delegate hosts that are ZooKeeper ensemble nodes. Delegate hosts that are not ZooKeeper ensemble nodes do not need the file.

Log in to a delegate host as root or as a user with superuser privileges.
Copy or move the install-zenoss-isvcs-zookeeper-v*.run file to /root.
Add execute permission to the file.
```
chmod +x /root/*.run
```

Update the master host

The following procedures are to be performed on the master host of a multi-host deployment or on the single host in a single-host deployment.

The procedures for updating a delegate host can be found below.

Stopping Control Center on the master host

Use this procedure to stop the Control Center service (serviced) on the master host.

Log in to the master host as root or as a user with superuser privileges.
Stop the top-level service serviced is managing, if necessary.
1. Show the status of running services.
```
serviced service status
```
  The top-level service is the service listed immediately below the headings line. - If the status of the top-level service and all child services is stopped, proceed to the next step. - If the status of the top-level service and all child services is not stopped, perform the remaining substeps.
2. Stop the top-level service.
```
serviced service stop Zenoss.resmgr
```
3. Monitor the stop.
```
serviced service status
```
  When the status of the top-level service and all child services is stopped, proceed to the next step.
Stop the Control Center service.
```
systemctl stop serviced
```
Ensure that no containers remain in the local repository.
1. Display the identifiers of all containers, running and exited.
```
docker ps -qa
```
  - If the command returns no result, stop. This procedure is complete.
  - If the command returns a result, perform the following substeps.
2. Remove all remaining containers.
```
docker ps -qa | xargs --no-run-if-empty docker rm -fv
```
3. Display the identifiers of all containers, running and exited.
```
docker ps -qa
```
  - If the command returns no result, stop. This procedure is complete.
  - If the command returns a result, perform the remaining substeps.
4. Disable the automatic startup of serviced.
```
systemctl disable serviced
```
5. Reboot the host.
```
reboot
```
6. Log in to the master host as root, or as a user with superuser privileges.
7. Enable the automatic startup of serviced.
```
systemctl enable serviced
```

Updating Docker to 18.09.6

Use this procedure to update Docker to version 18.09.6.

Log in as root or as a user with superuser privileges.
Determine which version of Docker is installed.
```
docker info 2>/dev/null | awk '/^ Server/ { print $3 }'
```
- If the result is 18.09.6, stop. This procedure is unnecessary; proceed to the next one.
- If the result is 17.09.1, perform the remaining steps in this procedure.
- If the result is 17.03.1, use the alternate procedure.

Update from 17.09.1Update from 17.03.1

1. Determine which release is installed.
```
cat /etc/redhat-release
```
  - If the result is greater than 7.2, proceed to the next step.
  - If the result is less than or equal to 7.1, perform the remaining substeps.
Update the operating system, if necessary.
1. Disable automatic start of serviced.
```
systemctl disable serviced
```
2. Update the operating system, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update && reboot
```
3. Log in as root, or as a user with superuser privileges.
4. Enable automatic start of serviced.
```
systemctl enable serviced
```
Update the Linux kernel, if necessary.
1. Determine which kernel version is installed.
```
uname -r
```
  If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the kernel, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update kernel && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Stop the Docker service.
```
systemctl stop docker
```
Remove Docker 17.09.0.
1. Remove without checking dependencies.
```
rpm -e --nodeps docker-ce
```
2. Clean the yum databases.
```
yum clean all
```
Install Docker CE 18.09.6.
```
yum install --enablerepo=zenoss-mirror docker-ce-18.09.6-3.el7
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Start the Docker service.
```
systemctl start docker
```

Remove the Docker repository description file, if present.

Upgrades no longer require external repositories.
```
rm -f /etc/yum.repos.d/docker.repo
```
Update the operating system, if necessary.
1. Determine which release is installed.
```
cat /etc/redhat-release
```
  - If the result is 7.2 or greater, proceed to the next step.
  - If the result is 7.1, perform the remaining substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the operating system, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Update the Linux kernel, if necessary.
1. Determine which kernel version is installed.
```
uname -r
```
  If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the kernel, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update kernel && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Stop the Docker service.
```
systemctl stop docker
```
Remove Docker 17.03.1.
1. Remove without checking dependencies.
```
rpm -e --nodeps docker-ce
```
2. Clean the yum databases.
```
yum clean all
```
Install Docker CE 18.09.6.
```
yum install --enablerepo=zenoss-mirror docker-ce-18.09.6-3.el7
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Start the Docker service.
```
systemctl start docker
```

Remove a previous workaround for an NFS rpcbind issue.

Remove the NFS service drop-in file, if it exists.

test -f /etc/systemd/system/nfs-server.service.d/nfs-server.conf \
  && rm -f /etc/systemd/system/nfs-server.service.d/nfs-server.conf

Reload the systemd manager configuration.
```
systemctl daemon-reload
```

Load image files

Use this procedure to load images into the local Docker registry on a host.

Log in to the host as root or as a user with superuser privileges.
Change directory to /root.
```
cd /root
```

Load the images.

for image in install-zenoss-*.run
do
  /bin/echo -en "\nLoading $image..."
  yes | ./$image
done

List the images in the registry.
```
docker images
```
The result should show one image for each archive file.
Optional: Delete the archive files, if desired.
```
rm -i ./install-zenoss-*.run
```

Update the serviced binary

Use this procedure to update the serviced binary on a Control Center host. Perform this procedure on each host in your Control Center deployment.

In multi-host deployments, stop serviced on the master host first.

Log in to the host as root or as a user with superuser privileges.

Start the docker service, if necessary.

systemctl is-active docker || systemctl start docker

Save the current serviced configuration file as a reference.
1. Rename the file.Replace VERSION with the new version number; for example, 1.7.0:
```
mv /etc/default/serviced /etc/default/serviced-pre-VERSION
```
2. Set permissions to read-only.
```
chmod 0440 /etc/default/serviced-pre-VERSION
```
Install the new serviced RPM package.Replace VERSION with the new version number; for example, 1.7.0:
```
yum install --enablerepo=zenoss-mirror /opt/zenoss-repo-mirror/serviced-VERSION-1.x86_64.rpm
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Make a backup copy of the new configuration file.
1. Copy the file.Replace VERSION with the new version number; for example, 1.7.0:
```
cp /etc/default/serviced /etc/default/serviced-VERSION-orig
```
2. Set permissions to read-only.
```
chmod 0440 /etc/default/serviced-VERSION-orig
```
Compare the new configuration file with the configuration file of the previous release.
1. Identify the configuration files to compare.
```
ls -l /etc/default/serviced*
```
  The original versions of the configuration files should end with orig, but you may have to compare the dates of the files.
2. Compare the new and previous configuration files.Replace New-Version with the name of the new configuration file, and replace Previous-Version with the name of the previous configuration file:
```
diff New-Version Previous-Version
```
  For example, to compare versions 1.6.3 and the most recent version (VERSION), enter the following command:
```
diff /etc/default/serviced-1.6.3-orig /etc/default/serviced-VERSION-orig
```
3. If the command returns no result, restore the backup of the previous configuration file.
```
cp /etc/default/serviced-pre-VERSION /etc/default/serviced && chmod 0644 /etc/default/serviced
```
4. If the command returns a result, restore the backup of the previous configuration file, and then optionally, use the results to edit the restored version.
```
cp /etc/default/serviced-pre-VERSION /etc/default/serviced && chmod 0644 /etc/default/serviced
```
  For more information about configuring a host, see Configuration variables.
Reload the systemd manager configuration.
```
systemctl daemon-reload
```
Single-host deployments only: Start the serviced service.
```
systemctl start serviced
```

Update the delegate hosts

Stopping Control Center on a delegate host

Use this procedure to stop the Control Center service (serviced) on a delegate host in a multi-host deployment. Repeat this procedure on each delegate host in your deployment.

Before performing this procedure on any delegate host, stop Control Center on the master host.

Log in to the delegate host as root or as a user with superuser privileges.
Stop the Control Center service.
```
systemctl stop serviced
```
Ensure that no containers remain in the local repository.
1. Display the identifiers of all containers, running and exited.
```
docker ps -qa
```
  - If the command returns no result, proceed to the next step.
  - If the command returns a result, perform the following substeps.
2. Remove all remaining containers.
```
docker ps -qa | xargs --no-run-if-empty docker rm -fv
```
  - If the remove command completes, proceed to the next step.
  - If the remove command does not complete, the most likely cause is an NFS conflict. Perform the following substeps.
3. Stop the NFS and Docker services.
```
systemctl stop nfs && systemctl stop docker
```
4. Start the NFS and Docker services.
```
systemctl start nfs && systemctl start docker
```
5. Repeat the attempt to remove all remaining containers.
```
docker ps -qa | xargs --no-run-if-empty docker rm -fv
```
  - If the remove command completes, proceed to the next step.
  - If the remove command does not complete, perform the remaining substeps.
6. Disable the automatic startup of serviced.
```
systemctl disable serviced
```
7. Reboot the host.
```
reboot
```
8. Log in to the delegate host as root, or as a user with superuser privileges.
9. Enable the automatic startup of serviced.
```
systemctl enable serviced
```
Dismount all filesystems mounted from the Control Center master host.

This step ensures no stale mounts remain when the storage on the master host is replaced.
1. Identify filesystems mounted from the master host.
```
awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
```
  - If the preceding command returns no result, stop. This procedure is complete.
  - If the preceding command returns a result, perform the following substeps.
2. Force the filesystems to dismount.
```
for FS in $(awk '/serviced/ { print $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs')
do
  umount -f $FS
done
```
3. Identify filesystems mounted from the master host.
```
awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
```
  - If the preceding command returns no result, stop. This procedure is complete.
  - If the preceding command returns a result, perform the following substeps.
4. Perform a lazy dismount.
```
for FS in $(awk '/serviced/ { print $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs')
do
  umount -f -l $FS
done
```
5. Restart the NFS service.
```
systemctl restart nfs
```
6. Determine whether any filesystems remain mounted from the master host.
```
awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
```
  - If the preceding command returns no result, stop. This procedure is complete.
  - If the preceding command returns a result, perform the remaining substeps.
7. Disable the automatic startup of serviced.
```
systemctl disable serviced
```
8. Reboot the host.
```
reboot
```
9. Log in to the delegate host as root, or as a user with superuser privileges.
10. Enable the automatic startup of serviced.
```
systemctl enable serviced
```

Updating Docker to 18.09.6

Use this procedure to update Docker to version 18.09.6.

Log in as root or as a user with superuser privileges.
Determine which version of Docker is installed.
```
docker info 2>/dev/null | awk '/^ Server/ { print $3 }'
```
- If the result is 18.09.6, stop. This procedure is unnecessary; proceed to the next one.
- If the result is 17.09.1, perform the remaining steps in this procedure.
- If the result is 17.03.1, use the alternate procedure.

Update from 17.09.1Update from 17.03.1

1. Determine which release is installed.
```
cat /etc/redhat-release
```
  - If the result is greater than 7.2, proceed to the next step.
  - If the result is less than or equal to 7.1, perform the remaining substeps.
Update the operating system, if necessary.
1. Disable automatic start of serviced.
```
systemctl disable serviced
```
2. Update the operating system, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update && reboot
```
3. Log in as root, or as a user with superuser privileges.
4. Enable automatic start of serviced.
```
systemctl enable serviced
```
Update the Linux kernel, if necessary.
1. Determine which kernel version is installed.
```
uname -r
```
  If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the kernel, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update kernel && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Stop the Docker service.
```
systemctl stop docker
```
Remove Docker 17.09.0.
1. Remove without checking dependencies.
```
rpm -e --nodeps docker-ce
```
2. Clean the yum databases.
```
yum clean all
```
Install Docker CE 18.09.6.
```
yum install --enablerepo=zenoss-mirror docker-ce-18.09.6-3.el7
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Start the Docker service.
```
systemctl start docker
```

Remove the Docker repository description file, if present.

Upgrades no longer require external repositories.
```
rm -f /etc/yum.repos.d/docker.repo
```
Update the operating system, if necessary.
1. Determine which release is installed.
```
cat /etc/redhat-release
```
  - If the result is 7.2 or greater, proceed to the next step.
  - If the result is 7.1, perform the remaining substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the operating system, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Update the Linux kernel, if necessary.
1. Determine which kernel version is installed.
```
uname -r
```
  If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the kernel, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update kernel && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Stop the Docker service.
```
systemctl stop docker
```
Remove Docker 17.03.1.
1. Remove without checking dependencies.
```
rpm -e --nodeps docker-ce
```
2. Clean the yum databases.
```
yum clean all
```
Install Docker CE 18.09.6.
```
yum install --enablerepo=zenoss-mirror docker-ce-18.09.6-3.el7
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Start the Docker service.
```
systemctl start docker
```

Remove a previous workaround for an NFS rpcbind issue.

Remove the NFS service drop-in file, if it exists.

test -f /etc/systemd/system/nfs-server.service.d/nfs-server.conf \
  && rm -f /etc/systemd/system/nfs-server.service.d/nfs-server.conf

Reload the systemd manager configuration.
```
systemctl daemon-reload
```

Update the serviced binary

Use this procedure to update the serviced binary on a Control Center host. Perform this procedure on each host in your Control Center deployment.

In multi-host deployments, stop serviced on the master host first.

Log in to the host as root or as a user with superuser privileges.

Start the docker service, if necessary.

systemctl is-active docker || systemctl start docker

Save the current serviced configuration file as a reference.
1. Rename the file.Replace VERSION with the new version number; for example, 1.7.0:
```
mv /etc/default/serviced /etc/default/serviced-pre-VERSION
```
2. Set permissions to read-only.
```
chmod 0440 /etc/default/serviced-pre-VERSION
```
Install the new serviced RPM package.Replace VERSION with the new version number; for example, 1.7.0:
```
yum install --enablerepo=zenoss-mirror /opt/zenoss-repo-mirror/serviced-VERSION-1.x86_64.rpm
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Make a backup copy of the new configuration file.
1. Copy the file.Replace VERSION with the new version number; for example, 1.7.0:
```
cp /etc/default/serviced /etc/default/serviced-VERSION-orig
```
2. Set permissions to read-only.
```
chmod 0440 /etc/default/serviced-VERSION-orig
```
Compare the new configuration file with the configuration file of the previous release.
1. Identify the configuration files to compare.
```
ls -l /etc/default/serviced*
```
  The original versions of the configuration files should end with orig, but you may have to compare the dates of the files.
2. Compare the new and previous configuration files.Replace New-Version with the name of the new configuration file, and replace Previous-Version with the name of the previous configuration file:
```
diff New-Version Previous-Version
```
  For example, to compare versions 1.6.3 and the most recent version (VERSION), enter the following command:
```
diff /etc/default/serviced-1.6.3-orig /etc/default/serviced-VERSION-orig
```
3. If the command returns no result, restore the backup of the previous configuration file.
```
cp /etc/default/serviced-pre-VERSION /etc/default/serviced && chmod 0644 /etc/default/serviced
```
4. If the command returns a result, restore the backup of the previous configuration file, and then optionally, use the results to edit the restored version.
```
cp /etc/default/serviced-pre-VERSION /etc/default/serviced && chmod 0644 /etc/default/serviced
```
  For more information about configuring a host, see Configuration variables.
Reload the systemd manager configuration.
```
systemctl daemon-reload
```
Single-host deployments only: Start the serviced service.
```
systemctl start serviced
```

Start Control Center

There are separate start-up procedures for single- and multi-host deployments of Control Center. Please use the procedure appropriate to your deployment.

Starting Control Center (single-host deployment)

Use this procedure to start Control Center in a single-host deployment. The default configuration of the Control Center service (serviced) is to start when the host starts. This procedure is only needed after stopping serviced to perform maintenance tasks.

Log in to the master host as root or as a user with superuser privileges.

Enable serviced, if necessary.

systemctl is-enabled serviced || systemctl enable serviced

Start the Control Center service.
```
systemctl start serviced
```
Optional: Monitor the startup, if desired.
```
journalctl -u serviced -f -o cat
```

Starting Control Center (multi-host deployment)

Use this procedure to start Control Center in a multi-host deployment. The default configuration of the Control Center service (serviced) is to start when the host starts. This procedure is only needed after stopping serviced to perform maintenance tasks.

Log in to the master host as root or as a user with superuser privileges.

Enable serviced, if necessary.

systemctl is-enabled serviced || systemctl enable serviced

Identify the hosts in the ZooKeeper ensemble.
```
grep -E '^[[:space:]]*SERVICED_ZK=' /etc/default/serviced
```
The result is a list of 1, 3, or 5 hosts, separated by the comma character (,). The master host is always a node in the ZooKeeper ensemble.
In separate windows, log in to each of the delegate hosts that are nodes in the ZooKeeper ensemble as root, or as a user with superuser privileges.
On all ensemble hosts, start serviced.

The window of time for starting a ZooKeeper ensemble is relatively short. The goal of this step is to start Control Center on each ensemble node at about the same time, so that each node can participate in electing the leader.
```
systemctl start serviced
```
On the master host, check the status of the ZooKeeper ensemble.
1. Attach to the container of the ZooKeeper service.
```
docker exec -it serviced-isvcs_zookeeper bash
```
2. Query the master host and identify its role in the ensemble.
  
  Replace Master with the hostname or IP address of the master host:
```
{ echo stats; sleep 1; } | nc Master 2181 | grep Mode
```
  The result includes leader or follower. When multiple hosts rely on the ZooKeeper instance on the master host, the result includes standalone.
3. Query the other delegate hosts to identify their role in the ensemble.
  
  Replace Delegate with the hostname or IP address of a delegate host:
```
{ echo stats; sleep 1; } | nc Delegate 2181 | grep Mode
```
4. Detach from the container of the ZooKeeper service.
```
exit
```
If none of the nodes reports that it is the ensemble leader within a few minutes of starting serviced, reboot the ensemble hosts.
Log in to each of the delegate hosts that are not nodes in the ZooKeeper ensemble as root, or as a user with superuser privileges, and then start serviced.
```
systemctl start serviced
```
Optional: Monitor the startup, if desired.
```
journalctl -u serviced -f -o cat
```

Post-upgrade procedures

Setting the connection timeout of a resource pool

Use this procedure to set the length of time the services scheduler waits for a disconnected delegate host to reconnect before moving its services to a different host in the resource pool. This affects all multi-host resource pools. Zenoss recommends using a minimum value of 15 seconds for all multi-host pools. For resource pools that are connected through high-latency, wide-area networks, Zenoss recommends values greater than 15 seconds.

Log in to the master host as a user with serviced CLI privileges.
Display the list of resource pools and their connection timeout values.
```
serviced pool list -v | grep -E 'ID|ConnectionTimeout'
```
Set the connection timeout value of a resource pool.

This command accepts the following units identifiers: - ms (milliseconds) - s (seconds) - m (minutes) - h (hours)

Replace Pool-ID with a resource pool identifier, and replace Timeout+Units with an integer followed by a units identifier:
```
serviced pool set-conn-timeout Pool-ID Timeout+Units
```

Updating from 1.4.1 or 1.4.2

The following procedures cover the upgrade of Control Center (versions 1.4.1 or 1.4.2) to 1.7.0.

If you are upgrading from 1.5.x or 1.6.3, please see Updating from 1.5.x or 1.6.3.

Outline

Download required files
Install the repository mirror
Stage Docker image files
Update the master host:
Update delegate hosts:
Start Control Center:
- Start Control Center (single-host deployment)
- Start Control Center (multi-host deployment)
Perform post-upgrade procedures:
- Set the connection timeout of a resource pool
- Remove unused images

Downloading Control Center files for release 1.7.0

Use this procedure to download required files to a workstation, and then copy the files to the hosts that need them.

To perform this procedure, you need:

A workstation with internet access.
Permission to download files from delivery.zenoss.io. Customers can request permission by filing a ticket at the Zenoss Support site.
A secure network copy program.

Follow these steps:

In a web browser, navigate to delivery.zenoss.io, and then log in.
Download the self-installing Docker image files.
- install-zenoss-serviced-isvcs:v63.run
- install-zenoss-isvcs-zookeeper:v11.run
Download the Control Center RPM file.
- serviced-1.7.0-1.x86_64.rpm
Identify the operating system release on Control Center hosts. Enter the following command on each Control Center host in your deployment, if necessary. All Control Center hosts should be running the same operating system release and kernel.
```
cat /etc/redhat-release
```
Download the RHEL/CentOS repository mirror file for your deployment. The download site provides repository mirror files containing the packages that Control Center requires. For RHEL/CentOS 7.7 and 7.8, use the 7.6 file.
- yum-mirror-centos7.2-1511-serviced-1.6.5.x86_64.rpm
- yum-mirror-centos7.3-1611-serviced-1.6.5.x86_64.rpm
- yum-mirror-centos7.4-1708-serviced-1.6.5.x86_64.rpm
- yum-mirror-centos7.5-1804-serviced-1.6.5.x86_64.rpm
- yum-mirror-centos7.6-1810-serviced-1.6.5.x86_64.rpm
  
  Currently, Red Hat Enterprise Linux (RHEL) 8.x and CentOS 8.x are unsupported.
Optional: Download the Zenoss Pretty Good Privacy (PGP) key, if desired. You can use the Zenoss PGP key to verify Zenoss RPM files and the yum metadata of the repository mirror.
1. Download the key.
```
curl --location -o /tmp/tmp.html 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xED0A5FD2AA5A1AD7'
```
2. Determine whether the download succeeded.
```
grep -Ec '^\-\-\-\-\-BEGIN PGP' /tmp/tmp.html
```
  - If the result is 0, return to the previous substep.
  - If the result is 1, proceed to the next substep.
3. Extract the key.
```
awk '/^-----BEGIN PGP.*$/,/^-----END PGP.*$/' /tmp/tmp.html > ./RPM-PGP-KEY-Zenoss
```
Use a secure copy program to copy the files to Control Center hosts.
- Copy all files to the master host.
- Copy the RHEL/CentOS RPM file, the Control Center RPM file, and the Zenoss PGP key file to all delegate hosts.
- Copy the Docker image file for ZooKeeper to delegate hosts that are ZooKeeper ensemble nodes.

Installing the repository mirror

Use this procedure to install the Zenoss repository mirror on a Control Center host. The mirror contains packages that are required on all Control Center hosts. Repeat this procedure on each host in your deployment.

Log in to the target host as root or as a user with superuser privileges.
Move the RPM files and the Zenoss PGP key file to /tmp.
Upgrades only: Remove the existing repository mirror, if necessary.
1. Search for the mirror.
```
yum list --disablerepo=* | awk '/^yum-mirror/ { print $1}'
```
2. Remove the mirror.
  
  Replace Old-Mirror with the name of the Zenoss repository mirror returned in the previous substep:
```
yum remove Old-Mirror
```
Install the repository mirror.
```
yum install /tmp/yum-mirror-*.rpm
```
The yum command copies the contents of the RPM file to /opt/zenoss-repo-mirror.
Optional: Install the Zenoss PGP key, and then test the package files, if desired.
1. Move the Zenoss PGP key to the mirror directory.
```
mv /tmp/RPM-PGP-KEY-Zenoss /opt/zenoss-repo-mirror
```
2. Install the key.
```
rpm --import /opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
```
3. Test the repository mirror package file.
```
rpm -K /tmp/yum-mirror-*.rpm
```
  On success, the result includes the file name and the following information:
```
(sha1) dsa sha1 md5 gpg OK
```
4. Test the Control Center package file.
```
rpm -K /tmp/serviced-VERSION-1.x86_64.rpm
```

Optional: Update the configuration file of the Zenoss repository mirror to enable PGP key verification, if desired.

Open the repository mirror configuration file (/etc/yum.repos.d/zenoss-mirror.repo) with a text editor, and then add the following lines to the end of the file.
```
repo_gpgcheck=1
gpgkey=file:///opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
```
Save the file, and then close the editor.

Update the yum metadata cache.

yum makecache fast

The cache update process includes the following prompt:

Retrieving key from file:///opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
Importing GPG key 0xAA5A1AD7:
 Userid     : "Zenoss, Inc. <dev@zenoss.com>"
 Fingerprint: f31f fd84 6a23 b3d5 981d a728 ed0a 5fd2 aa5a 1ad7
 From       : /opt/zenoss-repo-mirror/RPM-PGP-KEY-Zenoss
Is this ok [y/N]:

Enter y.

Move the Control Center package file to the mirror directory.

mv /tmp/serviced-VERSION-1.x86_64.rpm /opt/zenoss-repo-mirror

Optional: Delete the mirror package file, if desired.
```
rm /tmp/yum-mirror-*.rpm
```

Staging Docker image files on the master host

Before performing this procedure, verify that approximately 640MB of temporary space is available on the file system where /root is located.

Use this procedure to copy Docker image files to the Control Center master host. The files are used when Docker is fully configured.

Log in to the master host as root or as a user with superuser privileges.
Copy or move the archive files to /root.
Add execute permission to the files.
```
chmod +x /root/*.run
```

Update the master host

The following procedures are to be performed on the master host of a multi-host deployment or on the single host in a single-host deployment.

The procedures for updating a delegate host can be found below.

Stopping Control Center on the master host

Use this procedure to stop the Control Center service (serviced) on the master host.

Log in to the master host as root or as a user with superuser privileges.
Stop the top-level service serviced is managing, if necessary.
1. Show the status of running services.
```
serviced service status
```
  The top-level service is the service listed immediately below the headings line. - If the status of the top-level service and all child services is stopped, proceed to the next step. - If the status of the top-level service and all child services is not stopped, perform the remaining substeps.
2. Stop the top-level service.
```
serviced service stop Zenoss.resmgr
```
3. Monitor the stop.
```
serviced service status
```
  When the status of the top-level service and all child services is stopped, proceed to the next step.
Stop the Control Center service.
```
systemctl stop serviced
```
Ensure that no containers remain in the local repository.
1. Display the identifiers of all containers, running and exited.
```
docker ps -qa
```
  - If the command returns no result, stop. This procedure is complete.
  - If the command returns a result, perform the following substeps.
2. Remove all remaining containers.
```
docker ps -qa | xargs --no-run-if-empty docker rm -fv
```
3. Display the identifiers of all containers, running and exited.
```
docker ps -qa
```
  - If the command returns no result, stop. This procedure is complete.
  - If the command returns a result, perform the remaining substeps.
4. Disable the automatic startup of serviced.
```
systemctl disable serviced
```
5. Reboot the host.
```
reboot
```
6. Log in to the master host as root, or as a user with superuser privileges.
7. Enable the automatic startup of serviced.
```
systemctl enable serviced
```

Updating Docker to 18.09.6

Use this procedure to update Docker to version 18.09.6.

Log in as root or as a user with superuser privileges.
Determine which version of Docker is installed.
```
docker info 2>/dev/null | awk '/^ Server/ { print $3 }'
```
- If the result is 18.09.6, stop. This procedure is unnecessary; proceed to the next one.
- If the result is 17.03.1, perform the remaining steps in this procedure.
- If the result is 17.09.1, use the alternate procedure.

Update from 17.03.1Update from 17.09.1

Remove the Docker repository description file, if present.

Upgrades no longer require external repositories.
```
rm -f /etc/yum.repos.d/docker.repo
```
Update the operating system, if necessary.
1. Determine which release is installed.
```
cat /etc/redhat-release
```
  - If the result is 7.2 or greater, proceed to the next step.
  - If the result is 7.1, perform the remaining substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the operating system, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Update the Linux kernel, if necessary.
1. Determine which kernel version is installed.
```
uname -r
```
  If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the kernel, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update kernel && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Stop the Docker service.
```
systemctl stop docker
```
Remove Docker 17.03.1.
1. Remove without checking dependencies.
```
rpm -e --nodeps docker-ce
```
2. Clean the yum databases.
```
yum clean all
```
Install Docker CE 18.09.6.
```
yum install --enablerepo=zenoss-mirror docker-ce-18.09.6-3.el7
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Start the Docker service.
```
systemctl start docker
```

Remove a previous workaround for an NFS rpcbind issue.

Remove the NFS service drop-in file, if it exists.

test -f /etc/systemd/system/nfs-server.service.d/nfs-server.conf \
  && rm -f /etc/systemd/system/nfs-server.service.d/nfs-server.conf

Reload the systemd manager configuration.
```
systemctl daemon-reload
```

1. Determine which release is installed.
```
cat /etc/redhat-release
```
  - If the result is greater than 7.2, proceed to the next step.
  - If the result is less than or equal to 7.1, perform the remaining substeps.
Update the operating system, if necessary.
1. Disable automatic start of serviced.
```
systemctl disable serviced
```
2. Update the operating system, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update && reboot
```
3. Log in as root, or as a user with superuser privileges.
4. Enable automatic start of serviced.
```
systemctl enable serviced
```
Update the Linux kernel, if necessary.
1. Determine which kernel version is installed.
```
uname -r
```
  If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the kernel, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update kernel && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Stop the Docker service.
```
systemctl stop docker
```
Remove Docker 17.09.0.
1. Remove without checking dependencies.
```
rpm -e --nodeps docker-ce
```
2. Clean the yum databases.
```
yum clean all
```
Install Docker CE 18.09.6.
```
yum install --enablerepo=zenoss-mirror docker-ce-18.09.6-3.el7
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Start the Docker service.
```
systemctl start docker
```

Load image files

Use this procedure to load images into the local Docker registry on a host.

Log in to the host as root or as a user with superuser privileges.
Change directory to /root.
```
cd /root
```

Load the images.

for image in install-zenoss-*.run
do
  /bin/echo -en "\nLoading $image..."
  yes | ./$image
done

List the images in the registry.
```
docker images
```
The result should show one image for each archive file.
Optional: Delete the archive files, if desired.
```
rm -i ./install-zenoss-*.run
```

Update the serviced binary

Use this procedure to update the serviced binary on a Control Center host. Perform this procedure on each host in your Control Center deployment.

In multi-host deployments, stop serviced on the master host first.

Log in to the host as root or as a user with superuser privileges.

Start the docker service, if necessary.

systemctl is-active docker || systemctl start docker

Save the current serviced configuration file as a reference.
1. Rename the file.Replace VERSION with the new version number; for example, 1.7.0:
```
mv /etc/default/serviced /etc/default/serviced-pre-VERSION
```
2. Set permissions to read-only.
```
chmod 0440 /etc/default/serviced-pre-VERSION
```
Install the new serviced RPM package.Replace VERSION with the new version number; for example, 1.7.0:
```
yum install --enablerepo=zenoss-mirror /opt/zenoss-repo-mirror/serviced-VERSION-1.x86_64.rpm
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Make a backup copy of the new configuration file.
1. Copy the file.Replace VERSION with the new version number; for example, 1.7.0:
```
cp /etc/default/serviced /etc/default/serviced-VERSION-orig
```
2. Set permissions to read-only.
```
chmod 0440 /etc/default/serviced-VERSION-orig
```
Compare the new configuration file with the configuration file of the previous release.
1. Identify the configuration files to compare.
```
ls -l /etc/default/serviced*
```
  The original versions of the configuration files should end with orig, but you may have to compare the dates of the files.
2. Compare the new and previous configuration files.Replace New-Version with the name of the new configuration file, and replace Previous-Version with the name of the previous configuration file:
```
diff New-Version Previous-Version
```
  For example, to compare versions 1.6.3 and the most recent version (VERSION), enter the following command:
```
diff /etc/default/serviced-1.6.3-orig /etc/default/serviced-VERSION-orig
```
3. If the command returns no result, restore the backup of the previous configuration file.
```
cp /etc/default/serviced-pre-VERSION /etc/default/serviced && chmod 0644 /etc/default/serviced
```
4. If the command returns a result, restore the backup of the previous configuration file, and then optionally, use the results to edit the restored version.
```
cp /etc/default/serviced-pre-VERSION /etc/default/serviced && chmod 0644 /etc/default/serviced
```
  For more information about configuring a host, see Configuration variables.
Reload the systemd manager configuration.
```
systemctl daemon-reload
```
Single-host deployments only: Start the serviced service.
```
systemctl start serviced
```

Update the delegate hosts

Stopping Control Center on a delegate host

Use this procedure to stop the Control Center service (serviced) on a delegate host in a multi-host deployment. Repeat this procedure on each delegate host in your deployment.

Before performing this procedure on any delegate host, stop Control Center on the master host.

Log in to the delegate host as root or as a user with superuser privileges.
Stop the Control Center service.
```
systemctl stop serviced
```
Ensure that no containers remain in the local repository.
1. Display the identifiers of all containers, running and exited.
```
docker ps -qa
```
  - If the command returns no result, proceed to the next step.
  - If the command returns a result, perform the following substeps.
2. Remove all remaining containers.
```
docker ps -qa | xargs --no-run-if-empty docker rm -fv
```
  - If the remove command completes, proceed to the next step.
  - If the remove command does not complete, the most likely cause is an NFS conflict. Perform the following substeps.
3. Stop the NFS and Docker services.
```
systemctl stop nfs && systemctl stop docker
```
4. Start the NFS and Docker services.
```
systemctl start nfs && systemctl start docker
```
5. Repeat the attempt to remove all remaining containers.
```
docker ps -qa | xargs --no-run-if-empty docker rm -fv
```
  - If the remove command completes, proceed to the next step.
  - If the remove command does not complete, perform the remaining substeps.
6. Disable the automatic startup of serviced.
```
systemctl disable serviced
```
7. Reboot the host.
```
reboot
```
8. Log in to the delegate host as root, or as a user with superuser privileges.
9. Enable the automatic startup of serviced.
```
systemctl enable serviced
```
Dismount all filesystems mounted from the Control Center master host.

This step ensures no stale mounts remain when the storage on the master host is replaced.
1. Identify filesystems mounted from the master host.
```
awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
```
  - If the preceding command returns no result, stop. This procedure is complete.
  - If the preceding command returns a result, perform the following substeps.
2. Force the filesystems to dismount.
```
for FS in $(awk '/serviced/ { print $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs')
do
  umount -f $FS
done
```
3. Identify filesystems mounted from the master host.
```
awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
```
  - If the preceding command returns no result, stop. This procedure is complete.
  - If the preceding command returns a result, perform the following substeps.
4. Perform a lazy dismount.
```
for FS in $(awk '/serviced/ { print $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs')
do
  umount -f -l $FS
done
```
5. Restart the NFS service.
```
systemctl restart nfs
```
6. Determine whether any filesystems remain mounted from the master host.
```
awk '/serviced/ { print $1, $2 }' < /proc/mounts | grep -v '/opt/serviced/var/isvcs'
```
  - If the preceding command returns no result, stop. This procedure is complete.
  - If the preceding command returns a result, perform the remaining substeps.
7. Disable the automatic startup of serviced.
```
systemctl disable serviced
```
8. Reboot the host.
```
reboot
```
9. Log in to the delegate host as root, or as a user with superuser privileges.
10. Enable the automatic startup of serviced.
```
systemctl enable serviced
```

Updating Docker to 18.09.6

Use this procedure to update Docker to version 18.09.6.

Log in as root or as a user with superuser privileges.
Determine which version of Docker is installed.
```
docker info 2>/dev/null | awk '/^ Server/ { print $3 }'
```
- If the result is 18.09.6, stop. This procedure is unnecessary; proceed to the next one.
- If the result is 17.03.1, perform the remaining steps in this procedure.
- If the result is 17.09.1, use the alternate procedure.

Update from 17.03.1Update from 17.09.1

Remove the Docker repository description file, if present.

Upgrades no longer require external repositories.
```
rm -f /etc/yum.repos.d/docker.repo
```
Update the operating system, if necessary.
1. Determine which release is installed.
```
cat /etc/redhat-release
```
  - If the result is 7.2 or greater, proceed to the next step.
  - If the result is 7.1, perform the remaining substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the operating system, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Update the Linux kernel, if necessary.
1. Determine which kernel version is installed.
```
uname -r
```
  If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the kernel, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update kernel && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Stop the Docker service.
```
systemctl stop docker
```
Remove Docker 17.03.1.
1. Remove without checking dependencies.
```
rpm -e --nodeps docker-ce
```
2. Clean the yum databases.
```
yum clean all
```
Install Docker CE 18.09.6.
```
yum install --enablerepo=zenoss-mirror docker-ce-18.09.6-3.el7
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Start the Docker service.
```
systemctl start docker
```

Remove a previous workaround for an NFS rpcbind issue.

Remove the NFS service drop-in file, if it exists.

test -f /etc/systemd/system/nfs-server.service.d/nfs-server.conf \
  && rm -f /etc/systemd/system/nfs-server.service.d/nfs-server.conf

Reload the systemd manager configuration.
```
systemctl daemon-reload
```

1. Determine which release is installed.
```
cat /etc/redhat-release
```
  - If the result is greater than 7.2, proceed to the next step.
  - If the result is less than or equal to 7.1, perform the remaining substeps.
Update the operating system, if necessary.
1. Disable automatic start of serviced.
```
systemctl disable serviced
```
2. Update the operating system, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update && reboot
```
3. Log in as root, or as a user with superuser privileges.
4. Enable automatic start of serviced.
```
systemctl enable serviced
```
Update the Linux kernel, if necessary.
1. Determine which kernel version is installed.
```
uname -r
```
  If the result is lower than 3.10.0-327.22.2.el7.x86_64, perform the following substeps.
2. Disable automatic start of serviced.
```
systemctl disable serviced
```
3. Update the kernel, and then restart the host.
  
  The following commands require internet access or a local mirror of operating system packages.
```
yum makecache fast && yum update kernel && reboot
```
4. Log in as root, or as a user with superuser privileges.
5. Enable automatic start of serviced.
```
systemctl enable serviced
```
Stop the Docker service.
```
systemctl stop docker
```
Remove Docker 17.09.0.
1. Remove without checking dependencies.
```
rpm -e --nodeps docker-ce
```
2. Clean the yum databases.
```
yum clean all
```
Install Docker CE 18.09.6.
```
yum install --enablerepo=zenoss-mirror docker-ce-18.09.6-3.el7
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Start the Docker service.
```
systemctl start docker
```

Configuring NFS 4.0

Use this procedure to configure NFS 4.0 on delegate hosts if the operating system release is 7.4 or a more recent release. There may be a file locking defect in NFS 4.1 with RHEL/CentOS 7.4.

Log in to the host as root or as a user with superuser privileges.
Determine which release is installed.
```
cat /etc/redhat-release
```
- If the result is 7.4 or greater, perform the remaining steps of this procedure.
- If the result includes 7.2 or 7.3, continue to the next procedure.
Change the NFS configuration file.
1. Open /etc/nfsmount.conf with a text editor.
2. Locate the Defaultvers directive.
3. Remove the number sign character (#) from the beginning of the line.
4. Change the value from 4 to 4.0.
  
  The line should appear as follows:
```
Defaultvers=4.0
```
5. Save the file, and then close the editor.
Restart the NFS server.
```
systemctl restart nfs-server
```

Update the serviced binary

Use this procedure to update the serviced binary on a Control Center host. Perform this procedure on each host in your Control Center deployment.

In multi-host deployments, stop serviced on the master host first.

Log in to the host as root or as a user with superuser privileges.

Start the docker service, if necessary.

systemctl is-active docker || systemctl start docker

Save the current serviced configuration file as a reference.
1. Rename the file.Replace VERSION with the new version number; for example, 1.7.0:
```
mv /etc/default/serviced /etc/default/serviced-pre-VERSION
```
2. Set permissions to read-only.
```
chmod 0440 /etc/default/serviced-pre-VERSION
```
Install the new serviced RPM package.Replace VERSION with the new version number; for example, 1.7.0:
```
yum install --enablerepo=zenoss-mirror /opt/zenoss-repo-mirror/serviced-VERSION-1.x86_64.rpm
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.
Make a backup copy of the new configuration file.
1. Copy the file.Replace VERSION with the new version number; for example, 1.7.0:
```
cp /etc/default/serviced /etc/default/serviced-VERSION-orig
```
2. Set permissions to read-only.
```
chmod 0440 /etc/default/serviced-VERSION-orig
```
Compare the new configuration file with the configuration file of the previous release.
1. Identify the configuration files to compare.
```
ls -l /etc/default/serviced*
```
  The original versions of the configuration files should end with orig, but you may have to compare the dates of the files.
2. Compare the new and previous configuration files.Replace New-Version with the name of the new configuration file, and replace Previous-Version with the name of the previous configuration file:
```
diff New-Version Previous-Version
```
  For example, to compare versions 1.6.3 and the most recent version (VERSION), enter the following command:
```
diff /etc/default/serviced-1.6.3-orig /etc/default/serviced-VERSION-orig
```
3. If the command returns no result, restore the backup of the previous configuration file.
```
cp /etc/default/serviced-pre-VERSION /etc/default/serviced && chmod 0644 /etc/default/serviced
```
4. If the command returns a result, restore the backup of the previous configuration file, and then optionally, use the results to edit the restored version.
```
cp /etc/default/serviced-pre-VERSION /etc/default/serviced && chmod 0644 /etc/default/serviced
```
  For more information about configuring a host, see Configuration variables.
Reload the systemd manager configuration.
```
systemctl daemon-reload
```
Single-host deployments only: Start the serviced service.
```
systemctl start serviced
```

Updating the ZooKeeper image on ensemble nodes

Use this procedure to install a new Docker image for ZooKeeper on ZooKeeper ensemble nodes.

Log in to the master host as root or as a user with superuser privileges.
Identify the hosts in the ZooKeeper ensemble.
```
grep -E '^[[:space:]]*SERVICED_ZK=' /etc/default/serviced
```
The result is a list of 3 or 5 hosts, separated by the comma character (,). The master host is always a node in the ZooKeeper ensemble.
Log in to a ZooKeeper ensemble node as root, or as a user with superuser privileges.
Change directory to /root.
```
cd /root
```

Extract the ZooKeeper image.

yes | ./install-zenoss-isvcs-zookeeper_v*.run

Optional: Delete the archive file.

rm -i ./install-zenoss-isvcs-zookeeper_v*.run

Repeat the preceding four steps on each delegate that is a node in the ZooKeeper ensemble.

Start Control Center

There are separate start-up procedures for single- and multi-host deployments of Control Center. Please use the procedure appropriate to your deployment.

Starting Control Center (single-host deployment)

Use this procedure to start Control Center in a single-host deployment. The default configuration of the Control Center service (serviced) is to start when the host starts. This procedure is only needed after stopping serviced to perform maintenance tasks.

Log in to the master host as root or as a user with superuser privileges.

Enable serviced, if necessary.

systemctl is-enabled serviced || systemctl enable serviced

Start the Control Center service.
```
systemctl start serviced
```
Optional: Monitor the startup, if desired.
```
journalctl -u serviced -f -o cat
```

Starting Control Center (multi-host deployment)

Use this procedure to start Control Center in a multi-host deployment. The default configuration of the Control Center service (serviced) is to start when the host starts. This procedure is only needed after stopping serviced to perform maintenance tasks.

Log in to the master host as root or as a user with superuser privileges.

Enable serviced, if necessary.

systemctl is-enabled serviced || systemctl enable serviced

Identify the hosts in the ZooKeeper ensemble.
```
grep -E '^[[:space:]]*SERVICED_ZK=' /etc/default/serviced
```
The result is a list of 1, 3, or 5 hosts, separated by the comma character (,). The master host is always a node in the ZooKeeper ensemble.
In separate windows, log in to each of the delegate hosts that are nodes in the ZooKeeper ensemble as root, or as a user with superuser privileges.
On all ensemble hosts, start serviced.

The window of time for starting a ZooKeeper ensemble is relatively short. The goal of this step is to start Control Center on each ensemble node at about the same time, so that each node can participate in electing the leader.
```
systemctl start serviced
```
On the master host, check the status of the ZooKeeper ensemble.
1. Attach to the container of the ZooKeeper service.
```
docker exec -it serviced-isvcs_zookeeper bash
```
2. Query the master host and identify its role in the ensemble.
  
  Replace Master with the hostname or IP address of the master host:
```
{ echo stats; sleep 1; } | nc Master 2181 | grep Mode
```
  The result includes leader or follower. When multiple hosts rely on the ZooKeeper instance on the master host, the result includes standalone.
3. Query the other delegate hosts to identify their role in the ensemble.
  
  Replace Delegate with the hostname or IP address of a delegate host:
```
{ echo stats; sleep 1; } | nc Delegate 2181 | grep Mode
```
4. Detach from the container of the ZooKeeper service.
```
exit
```
If none of the nodes reports that it is the ensemble leader within a few minutes of starting serviced, reboot the ensemble hosts.
Log in to each of the delegate hosts that are not nodes in the ZooKeeper ensemble as root, or as a user with superuser privileges, and then start serviced.
```
systemctl start serviced
```
Optional: Monitor the startup, if desired.
```
journalctl -u serviced -f -o cat
```

Post-upgrade procedures

Setting the connection timeout of a resource pool

Use this procedure to set the length of time the services scheduler waits for a disconnected delegate host to reconnect before moving its services to a different host in the resource pool. This affects all multi-host resource pools. Zenoss recommends using a minimum value of 15 seconds for all multi-host pools. For resource pools that are connected through high-latency, wide-area networks, Zenoss recommends values greater than 15 seconds.

Log in to the master host as a user with serviced CLI privileges.
Display the list of resource pools and their connection timeout values.
```
serviced pool list -v | grep -E 'ID|ConnectionTimeout'
```
Set the connection timeout value of a resource pool.

This command accepts the following units identifiers: - ms (milliseconds) - s (seconds) - m (minutes) - h (hours)

Replace Pool-ID with a resource pool identifier, and replace Timeout+Units with an integer followed by a units identifier:
```
serviced pool set-conn-timeout Pool-ID Timeout+Units
```

Removing unused images

Use this procedure to identify and remove unused Control Center images.

Log in to the master host as root or as a user with superuser privileges.

Identify the images associated with the installed version of serviced.

serviced version | grep Images

Example result:

IsvcsImages: [zenoss/serviced-isvcs:v73 zenoss/isvcs-zookeeper:v16]

Start Docker, if necessary.

systemctl status docker || systemctl start docker

Display the serviced images in the local repository.

docker images | awk '/REPO|isvcs/'

Example result (edited to fit):

REPOSITORY                     TAG       IMAGE ID
zenoss/serviced-isvcs          v73       0bd933b3fb2f
zenoss/serviced-isvcs          v70       c19f1e317158
zenoss/isvcs-zookeeper         v16       7ea8c92ca1ad
zenoss/isvcs-zookeeper         v14       0ff3b3117fb8

The example result shows the current versions and one set of previous versions. Your result may include additional previous versions and will show different images IDs.

Remove unused images.

Replace Image-ID with the image ID of an image for a previous version.
```
docker rmi Image-ID
```
Repeat this command for each unused image.