Skip to content

Troubleshooting 6.8.0 upgrades

The following procedures can be used to correct issues with your Resource Manager 6.8.0 upgrade.

Docker image tags

Early versions of the zenoss/resmgr_6.8 Docker image shipped with an error in one of the upgrade migration scripts. Specifically, the migrate-services.txt file contains references to pre-release versions of Resource Manager Docker images.

During the execution of the upgrade-resmgr.sh script, the migrate-services.txt script is called to pass the tagged image names to Docker for import. When the Docker service can't find images matching these tags, it may show the following error message:

pull access denied for zenoss/resmgr_6.8, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

These errors can be found on lines 34 and 37 of the migrate-services.txt file:

/root/6.8.x/migrate-services.txt
28
29
30
31
32
33
34
35
36
37
38
39
40
DESCRIPTION Zenoss Service Dynamics 6.8.0 WITH_IMPACT Service Migrations
VERSION resmgr-6.8.0
REQUIRE_SVC
SNAPSHOT preupgrade-resmgr-6.8.0

# Choose images to upgrade to
SVC_USE zenoss/resmgr_6.8:6.8.0_1834_unstable zenoss/resmgr_5.0 zenoss/resmgr_5.1 zenoss/resmgr_5.2 zenoss/resmgr_5.3 zenoss/resmgr_6.0 zenoss/resmgr_6.1 zenoss/resmgr_6.2 zenoss/resmgr_6.3 zenoss/resmgr_6.4 zenoss/resmgr_6.5 zenoss/resmgr_6.6 zenoss/resmgr_6.7
SVC_USE zenoss/hbase:24.0.10
SVC_USE zenoss/opentsdb:24.0.10
SVC_USE zenoss/mariadb-resmgr:6.8.0_1834_unstable zenoss/mariadb
SVC_USE zenoss/memcached:cc38f31
SVC_USE zenoss/redis:760f98c
UPGRADE_RABBITMQ_IMAGE

The zenoss/resmgr_6.8 and zenoss/mariadb-resmgr images are followed with the 6.8.0_1834_unstable tag. However, these images should be followed with the latest tag, as shown in the following example:

/root/6.8.x/migrate-services.txt
28
29
30
31
32
33
34
35
36
37
38
39
40
DESCRIPTION Zenoss Service Dynamics 6.8.0 WITH_IMPACT Service Migrations
VERSION resmgr-6.8.0
REQUIRE_SVC
SNAPSHOT preupgrade-resmgr-6.8.0

# Choose images to upgrade to
SVC_USE zenoss/resmgr_6.8:latest zenoss/resmgr_5.0 zenoss/resmgr_5.1 zenoss/resmgr_5.2 zenoss/resmgr_5.3 zenoss/resmgr_6.0 zenoss/resmgr_6.1 zenoss/resmgr_6.2 zenoss/resmgr_6.3 zenoss/resmgr_6.4 zenoss/resmgr_6.5 zenoss/resmgr_6.6 zenoss/resmgr_6.7
SVC_USE zenoss/hbase:24.0.10
SVC_USE zenoss/opentsdb:24.0.10
SVC_USE zenoss/mariadb-resmgr:latest zenoss/mariadb
SVC_USE zenoss/memcached:cc38f31
SVC_USE zenoss/redis:760f98c
UPGRADE_RABBITMQ_IMAGE

You can edit these lines manually or use the following procedure.

Back up, edit, and restart the upgrade script

  1. Make a backup copy of the migrate-services.txt file.

    cp /root/6.8.x/migrate-services.txt /root/6.8.x/migrate-services.txt.orig

  2. Run the following sed commands to edit the file.

    sed -i 's/SVC_USE zenoss\/resmgr_6\.8:6\.8\.0_1834_unstable/SVC_USE zenoss\/resmgr_6\.8:latest/' /root/6.8.x/migrate-services.txt
sed -i 's/SVC_USE zenoss\/mariadb-resmgr:6\.8\.0_1834_unstable/SVC_USE zenoss\/mariadb-resmgr:latest/' /root/6.8.x/migrate-services.txt

  3. Restart the upgrade script.

    /root/6.8.x/upgrade-resmgr.sh

Solr fails during upgrade

When upgrading to ZSD 6.8, your upgrade might fail with the following exception log message:

Exception writing document id /zport/ZenPortletManager/portlets/ChassisCapacity to the index

In this case, you need to remove Solr export and import commands in the script so you can rebuild the catalog.

To resolve this error, perform the following steps:

  1. Roll back the failed upgrade. For this fix, be sure to use the backup of Resource Manager that you created when preparing to update.

  2. Navigate to the /root/6.8.x/ directory.

  3. Run the following sed commands to edit the upgrade-resmgr.sh file.

    sed -i '/Export Solr index/,/solr_docs_8x.json$/d' upgrade-resmgr.sh
sed -i '/Import Solr index/,/--deleteAll$/d' upgrade-resmgr.sh
sed -i '/Removing collector call queues/ i\
    echo "Check Solr is started" \
    serviced service attach ${SOLR_SERVICE} bash -c "until curl -sI http://localhost:8983/solr/admin/cores | grep -q 200; do sleep 5; done" \
    echo "Removing Solr collection zenoss_model" \
    serviced service attach ${SOLR_SERVICE} bash -c "curl '\''http://localhost:8983/solr/admin/collections?action=DELETE&name=zenoss_model'\''" \
    sleep 120 \
    echo "Recreating Solr collection zenoss_model with one shard" \
    serviced service attach ${SOLR_SERVICE} bash -c "curl '\''http://localhost:8983/solr/admin/collections?action=CREATE&name=zenoss_model&collection.configName=zenoss_model&numShards=1&shards=shard1'\''" \
    echo "Full rebuilding of zenoss catalog" \
    serviced service shell -i zope su - zenoss -c "zencatalog run --createcatalog --forceindex --workers=4" \

' upgrade-resmgr.sh

When completed, restart the upgrade process. See Update Resource Manager.

Warning

As part of this modified procedure, Zenoss performs a complete rebuild of the Solr index. Depending on the size of your ZSD instance, this rebuild might take a significant amount of time, from a few minutes to multiple hours. Contact your Zenoss Support representative for more information about your instance.