Troubleshooting 6.9.0 upgrades
The following procedures can be used to correct issues with your Resource Manager 6.9.0 upgrade.
Docker configuration error causes appliance upgrade failure
Appliance images prior to 6.9.0 contained a python error that caused a broken docker configuration in certain circumstances. If this occurs, the upgrade will fail with the following error:
INFO:zen.migrate:Loading Reports
2025-06-19 15:09:27 INFO zen.migrate: Loading Reports
Ver Name Status
--------+----------------------------+-------
200.7.0 ZensyslogSvcDevForMsgParsing OK
200.7.0 ZensyslogSvcDevForMsgParsing OK
200.7.0 ZentrapSvcDevForMsgParsing OK
200.7.0 addConfigCacheProperties OK
200.7.0 addzNoRelationshipCopy OK
2025-06-19 15:09:29 INFO zen.migrate: Committing changes
2025-06-19 15:09:29,272 INFO zen.migrate: Committing changes
2025-06-19 15:09:29 INFO zen.migrate: Migration successful
2025-06-19 15:09:29,300 INFO zen.migrate: Migration successful
2025/06/19 15:09:33.161389 tls.go:200: WARN SSL/TLS verifications disabled.
FATA[0105] Unable to commit container containerid=47d496b0c965b8b45ac7e981012f44e71bbdaed54e06524515456c72299b6c85 containername=1x04xFK3YPdgt9LtR9Lgy1 error=registry index: image not found location=service.go:2106 logger=cli.api
ERRO[0235] Unable to execute step command=SVC_RUN error=exit status 1 line=SVC_RUN "Zenoss.resmgr/Zenoss/User Interface/Zope" upgrade location=runner.go:140 logger=script step=14
exit status 1
Script done.
If this error occurs, you will need to rerun the upgrade after modifying the
/etc/sysconfig/docker
file to repair the --insecure-registry
option by removing
the letter b
from in front of the IP address. For example, locate the following code:
OPTIONS="-s devicemapper --storage-opt dm.basesize=45G --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.mountopt=discard --exec-opt native.cgroupdriver=cgroupfs --log-level=error --insecure-registry b192.168.93.228:5000 --dns=172.17.0.1 --bip=172.17.0.1/24"
Then change this snippet to the following code with the b
removed from the IP address:
OPTIONS="-s devicemapper --storage-opt dm.basesize=45G --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.mountopt=discard --exec-opt native.cgroupdriver=cgroupfs --log-level=error --insecure-registry 192.168.93.228:5000 --dns=172.17.0.1 --bip=172.17.0.1/24"
Missing monitoring templates on new installs
In some cases, the model catalog may be improperly initialized and missing entries for some of the base objects in the product. You can detect if base objects are missing by navigating to Advanced->Monitoring Templates and verifying that no templates are listed.
To resolve this issue, rebuild the catalog with the following command that you can run from the CC master host:
serviced service run zope rebuild-catalog
Device monitoring fails with connection or DNS errors
On some versions of RHEL and related Linux distributions, the DNS configuration used for Docker is not required and causes DNS failures. You can see this error when monitoring an AWS or Azure device, for example:
AWS:
During processing 1 from 1 datapoints in EC2Instance/NetworkOut an error occured: Could not connect to the endpoint URL: "https://monitoring.us-east-1.amazonaws.com/"
Azure:
Unexpected error: DNSLookupError: DNS lookup failed: Couldn't find the hostname 'login.microsoftonline.com'
To resolve this issue, remove the --dns
option from the /etc/sysconf/docker
file.
Locate the following code:
OPTIONS="-s devicemapper --storage-opt dm.basesize=45G --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.mountopt=discard --exec-opt native.cgroupdriver=cgroupfs --log-level=error --insecure-registry 10.88.122.5:5000 --dns=172.17.0.1 --bip=172.17.0.1/24"
Then change this snippet to the following code with the --dns
option removed:
OPTIONS="-s devicemapper --storage-opt dm.basesize=45G --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.mountopt=discard --exec-opt native.cgroupdriver=cgroupfs --log-level=error --insecure-registry 10.88.122.5:5000 --bip=172.17.0.1/24"
Warning message about RabbitMQ running on a delegate host
When upgrading multi-host deployments, an upgrade step to remove an unused RabbitMQ queue might not run properly and a warning message is sent to the upgrade log, for example:
WARNING. RabbitMQ is running on a delegate host, to remove deprecated queues from exchange run the command on that host.
If you encounter this warning, follow the instructions and run the provided command on the delegate host that is currently running the RabbitMQ service.