Skip to content

Troubleshooting 6.9.0 upgrades

The following procedures can be used to correct issues with your Resource Manager 6.9.0 upgrade.

Docker configuration error causes appliance upgrade failure

Appliance images prior to 6.9.0 contained a python error that caused a broken docker configuration in certain circumstances. If this occurs, the upgrade will fail with the following error:

INFO:zen.migrate:Loading Reports
2025-06-19 15:09:27 INFO zen.migrate: Loading Reports
 Ver     Name                         Status
--------+----------------------------+-------
200.7.0  ZensyslogSvcDevForMsgParsing  OK      
200.7.0  ZensyslogSvcDevForMsgParsing  OK      
200.7.0  ZentrapSvcDevForMsgParsing    OK      
200.7.0  addConfigCacheProperties      OK      
200.7.0  addzNoRelationshipCopy        OK      
2025-06-19 15:09:29 INFO zen.migrate: Committing changes
2025-06-19 15:09:29,272 INFO zen.migrate: Committing changes
2025-06-19 15:09:29 INFO zen.migrate: Migration successful
2025-06-19 15:09:29,300 INFO zen.migrate: Migration successful
2025/06/19 15:09:33.161389 tls.go:200: WARN SSL/TLS verifications disabled.
FATA[0105] Unable to commit container                    containerid=47d496b0c965b8b45ac7e981012f44e71bbdaed54e06524515456c72299b6c85 containername=1x04xFK3YPdgt9LtR9Lgy1 error=registry index: image not found location=service.go:2106 logger=cli.api
ERRO[0235] Unable to execute step                        command=SVC_RUN error=exit status 1 line=SVC_RUN "Zenoss.resmgr/Zenoss/User Interface/Zope" upgrade location=runner.go:140 logger=script step=14
exit status 1
Script done.

If this error occurs, you will need to rerun the upgrade after modifying the /etc/sysconfig/docker file to repair the --insecure-registry option by removing the letter b from in front of the IP address. For example, locate the following code:

OPTIONS="-s devicemapper --storage-opt dm.basesize=45G --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.mountopt=discard --exec-opt native.cgroupdriver=cgroupfs --log-level=error --insecure-registry b192.168.93.228:5000 --dns=172.17.0.1 --bip=172.17.0.1/24"

Then change this snippet to the following code with the b removed from the IP address:

OPTIONS="-s devicemapper --storage-opt dm.basesize=45G --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.mountopt=discard --exec-opt native.cgroupdriver=cgroupfs --log-level=error --insecure-registry 192.168.93.228:5000 --dns=172.17.0.1 --bip=172.17.0.1/24"

Missing monitoring templates on new installs

In some cases, the model catalog may be improperly initialized and missing entries for some of the base objects in the product. You can detect if base objects are missing by navigating to Advanced->Monitoring Templates and verifying that no templates are listed.

To resolve this issue, rebuild the catalog with the following command that you can run from the CC master host:

serviced service run zope rebuild-catalog

Device monitoring fails with connection or DNS errors

On some versions of RHEL and related Linux distributions, the DNS configuration used for Docker is not required and causes DNS failures. You can see this error when monitoring an AWS or Azure device, for example:

AWS: During processing 1 from 1 datapoints in EC2Instance/NetworkOut an error occured: Could not connect to the endpoint URL: "https://monitoring.us-east-1.amazonaws.com/"

Azure: Unexpected error: DNSLookupError: DNS lookup failed: Couldn't find the hostname 'login.microsoftonline.com'

To resolve this issue, remove the --dns option from the /etc/sysconf/docker file.

Locate the following code:

OPTIONS="-s devicemapper --storage-opt dm.basesize=45G --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.mountopt=discard --exec-opt native.cgroupdriver=cgroupfs --log-level=error --insecure-registry 10.88.122.5:5000 --dns=172.17.0.1 --bip=172.17.0.1/24"

Then change this snippet to the following code with the --dns option removed:

OPTIONS="-s devicemapper --storage-opt dm.basesize=45G --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.mountopt=discard --exec-opt native.cgroupdriver=cgroupfs --log-level=error --insecure-registry 10.88.122.5:5000 --bip=172.17.0.1/24"

Warning message about RabbitMQ running on a delegate host

When upgrading multi-host deployments, an upgrade step to remove an unused RabbitMQ queue might not run properly and a warning message is sent to the upgrade log, for example:

WARNING. RabbitMQ is running on a delegate host, to remove deprecated queues from exchange run the command on that host.

If you encounter this warning, follow the instructions and run the provided command on the delegate host that is currently running the RabbitMQ service.