Skip to content

Updating non-appliance deployments to release 6.8.0

Use the instructions on this page to update Resource Manager deployments that are not based on a virtual appliance. The instructions on this page support updates from release 6.7.0 to 6.8.0.

To perform some of the procedures on this page, you need:

  • A workstation with internet access.
  • Permission to download files from delivery.zenoss.io. Customers can request permission by filing a ticket at the Zenoss Support site.
  • A secure network copy program.

Upgrade Control Center first

These instructions do not include steps for updating Control Center or the host operating system, and Control Center must be updated to release 1.11.0 first. For more information, see the Control Center documentation.

HBase Migration - Potential Data Loss

The Resource Manager 6.8.0 upgrade includes a migration to a new version of Apache HBase. Due to format changes in the underlying tables, historical performance data cannot be migrated directly. To preserve monitoring data, it must first be exported, then re-imported into HBase.

Read and understand the HBase migration tasks before you begin your installation.

Prepare to update

Verify normal operation, create backup

Follow these steps to prepare your deployment:

  1. Verify that Resource Manager is operating normally.

    1. Log in to the Control Center browser interface as ccuser.

    2. In the Application column of the Applications table, click Zenoss.resmgr.

    3. Verify that all of the services in the IP Assignments table have valid IP addresses.

    4. Scroll down to the Services table, and then verify that none of the child services have failing health checks.

  2. Check the integrity of Resource Manager databases. For more information, see Using Zenoss Toolbox.

  3. Create a backup of Resource Manager with the Control Center backup feature.

  4. If you plan to retain historical performance data, as detailed in the HBase migration tasks, you will need to separately back up the HBase folders.

    As the root user, run the following commands on the Control Center master host. Make sure you have adequate disk space before beginning.

    TENANT_ID=$(serviced service status Zenoss.resmgr | sed -n '2p' | awk {'print $2'})

cd /opt/serviced/var/volumes/$TENANT_ID

tar -czf hbase-master.tar.gz hbase-master/
tar -czf hbase-zk1.tar.gz hbase-zookeeper-1/
tar -czf hbase-zk2.tar.gz hbase-zookeeper-2/
tar -czf hbase-zk3.tar.gz hbase-zookeeper-3/

    Retain the HBase backups until you have confirmed that they have imported correctly.

Install the base image, if necessary

The Resource Manager update script requires content from the base image of the installed release. If the layers of the base image were flattened after installation, the image may not have the expected label, and the upgrade script will not find it.

Follow these steps to determine whether the base image is present, and to download and import it, if necessary:

  1. Log in to the Control Center host as root or as a user with superuser privileges.

  2. Search for the base image.

    docker images | grep -E 'zenoss/resmgr'

    • If the command returns a result, the base image is present. Continue to the next procedure.

    • If the command returns no result, the base image is absent. Perform the remaining steps of this procedure.

  3. In a web browser, navigate to delivery.zenoss.io, and then log in.

  4. Download the self-installing base image file for the installed release of Resource Manager.

    install-zenoss-resmgr_6.7_6.7.0_1.run

  5. Copy the file to the Control Center master host.

  6. Log in to the master host as root or as a user with superuser privileges.

  7. Change directory to the directory where the image file is located.

  8. Add execute permission to the image file and then import it.

    chmod +x ./install-zenoss-resmgr*run && yes | ./install-zenoss-resmgr*run

  9. Verify the base image is present.

    docker images | grep -E 'zenoss/resmgr'

Download and import Resource Manager image files

Use this procedure to download files required to update Resource Manager and then to copy the files to a Control Center master host.

Follow these steps:

  1. In a web browser, navigate to delivery.zenoss.io, and then log in.

  2. Download the self-installing Docker image files for Resource Manager 6.8.0.

    install-zenoss-resmgr_6.8_6.8.0_1.run
install-zenoss-mariadb-resmgr_6.8.0_1.run
install-zenoss-hbase_24.0.10.run
install-zenoss-opentsdb_24.0.10.run
install-zenoss-memcached_cc38f31.run
install-zenoss-redis_760f98c.run
install-zenoss-rabbitmq_5ccc9e3.run

  3. Copy the files to the Control Center master host.

  4. Log in to the master host as root or as a user with superuser privileges.

  5. Copy or move the archive files to /root.

  6. Add execute permission to the files.

    chmod +x /root/*.run

  7. Change directory to /root.

    cd /root

  8. Import the images.

    for image in install-zenoss-*.run
do
  /bin/echo -en "\nLoading $image..."
  yes | ./$image
done

  9. List the images in the registry.

    docker images

    The result should include one image for each archive file.

  10. Optional: Delete the archive files, if desired.

    rm -i ./install-*.run

  11. Copy the update scripts from the new Resource Manager image to a directory in /root.

    docker run -it --rm -v /root:/mnt/root zenoss/resmgr_6.8:6.8.0_1 rsync -a /root/6.7.x /mnt/root

Optional: Manage custom MariaDB passwords

If you are using custom database passwords for the mariadb-events and mariadb-model services, the passwords must be managed before starting the update process for Resource Manager.

Follow these steps to determine whether custom passwords are in use:

  1. Log in to the Control Center master host as a user with serviced CLI privileges.

  2. Display the MariaDB passwords.

    serviced service list Zenoss.resmgr | awk '/(zep|zodb).*-password/'

    When the default passwords are in use, the result looks like this:

         "global.conf.zep-admin-password": "",
     "global.conf.zep-password": "zenoss",
     "global.conf.zodb-admin-password": "",
     "global.conf.zodb-password": "zenoss",

You can manage the passwords with either of the following methods:

Perform the update

Stop Resource Manager

Follow these steps:

  1. Log in to the Control Center master host as a user with serviced CLI privileges.

  2. Check the status of Resource Manager.

    serviced service status Zenoss.resmgr --show-fields 'Name,ServiceID,Status'

    • If the status of all services is stopped, this procedure is complete. Continue to the next procedure.

    • If the status is running, perform the remaining steps.

  3. Stop Resource Manager.

    serviced service stop Zenoss.resmgr

  4. Check the status of Resource Manager.

    serviced service status Zenoss.resmgr --show-fields 'Name,ServiceID,Status'

    Repeat the command until the status of all services is stopped.

Update Resource Manager

Follow these steps to update Resource Manager.

  1. Log in to the Control Center master host as root or as a user with superuser privileges.

  2. Start the update script.

    /root/6.7.x/upgrade-resmgr.sh

    The update process begins. If you encounter errors, see Common update error recovery procedures.

    The following message appears frequently and can be ignored:

    sync.go:85: ERR Failed to publish events caused by: EOF

  3. Stop Resource Manager. Some Resource Manager services are started during the update, and they must be stopped.

    serviced service stop Zenoss.resmgr

HBase migration tasks

The Resource Manager 6.8.0 upgrade includes a migration to a new version of Apache HBase. Due to format changes in the underlying tables, historical performance data cannot be migrated directly. To preserve monitoring data, it must first be exported, then re-imported into HBase.

There are two options for performing the HBase migration:

Direct upgrade with data loss
If you do not need to retain historical performance data, you can delete the HBase tables. New, empty tables will be created when you start the Resource Manager application.
Export, upgrade, then import

If you need to retain historical performance data, you can save backups of the HBase folders before you begin the upgrade. Backup steps are detailed above in the Verify normal operation, create backup section.

These backups can then be imported into the upgraded HBase cluster, preserving previously gathered metric data.

These steps are separated into two phases to allow administrators to upgrade Resource Manager and import metrics separately.

Read and understand the following options before you begin your installation.

Direct upgrade with data loss

The following procedure will result in complete loss of all historical performance data and is not recommended for production Resource Manager installations.

  1. Perform the upgrade steps detailed in Perform the update.

  2. Confirm that the HBase cluster and OpenTSDB services are stopped.

    serviced service status Hbase
serviced service status opentsdb

    If either service is running, stop it.

    serviced service stop Hbase
serviced service stop opentsdb

  3. Remove all folders related to the HBase cluster.

    TENANT_ID=$(serviced service status Zenoss.resmgr | sed -n '2p' | awk {'print $2'})
cd /opt/serviced/var/volumes/$TENANT_ID
rm -rf hbase-master/ hbase-zookeeper-*

  4. Proceed to Perform post-update tasks.

Export, upgrade, then import

The following procedure will preserve historical performance data and is intended for production Resource Manager instances.

The time required for a complete import of all metric data is highly variable, depending on the amount of data, CPU, RAM, and storage performance.

If you don't have time to upgrade to Resource Manager 6.8.0 and import metrics in a single session, you can complete Phase 1 and begin using your upgraded instance. You can return to complete Phase 2 later.

Phase 1 - Upgrade and HBase image rollback

The first phase of this procedure rolls back the HBase and OpenTSDB services to use the Docker images included in Resource Manager 6.7.0.

  1. Confirm that you have performed the HBase backup step detailed in the Verify normal operation, create backup section above.

  2. Perform the upgrade steps detailed in Perform the update.

  3. Confirm that the HBase cluster and OpenTSDB services are stopped.

    serviced service status Hbase
serviced service status opentsdb

    If either service is running, stop it.

    serviced service stop Hbase
serviced service stop opentsdb

  4. Roll back the HBase and OpenTSDB services to use their Resource Manager 6.7.0 Docker images.

    TENANT_ID=$(serviced service status Zenoss.resmgr | sed -n '2p' | awk {'print $2'})
serviced docker override localhost:5000/$TENANT_ID/hbase:latest zenoss/hbase:24.0.9
serviced docker override localhost:5000/$TENANT_ID/opentsdb:latest zenoss/hbase:24.0.9
serviced docker sync

If you plan to use your Resource Manager 6.8.0 instance immediately, proceed to Perform post-update tasks. You may return to perform Phase 2 later.

If you plan to import metrics immediately, proceed to Phase 2 - Upgrade and metric import.

Phase 2 - Upgrade and metric import

The second phase of this procedure imports historical performance data into new, empty HBase tables.

  1. Stop all Resource Manager services except HBase.

    1. Stop Resource Manager.

      serviced service stop Zenoss.resmgr

      To confirm it has stopped, you may use:

      serviced service status Zenoss.resmgr

    2. Start the HBase services.

      serviced service start Hbase

  2. Check HBase for errors and fix them if found.

    serviced service attach hmaster su - hbase -c "JAVA_HOME=/usr/lib/jvm/jre /opt/hbase/bin/hbase hbck"

  3. Attach to the HBase container and export all tables to a separate folder.

    1. Attach to the Hbase container.

      serviced service attach hmaster su - hbase

    2. Export tables.

      HBASE_HOME=/opt/hbase
EXPORT_DIR=/var/hbase/hdfsTableExport
MAPRED_JAVA_PATH=org.apache.hadoop.hbase.mapreduce
mkdir -p $EXPORT_DIR
START=$(date +%s)
for current_table in tsdb tsdb-meta tsdb-tree tsdb-uid
do
    echo "$current_table export started"
    $HBASE_HOME/bin/hbase $MAPRED_JAVA_PATH.Export $current_table $EXPORT_DIR/$current_table 2>&1 | tee -a $EXPORT_DIR/hbase-export.log | grep mapreduce.Job
    echo "$current_table export finished"
    echo "$CONTROLPLANE_TENANT_ID-$current_table export started"
    $HBASE_HOME/bin/hbase $MAPRED_JAVA_PATH.Export $CONTROLPLANE_TENANT_ID-$current_table $EXPORT_DIR/$CONTROLPLANE_TENANT_ID-$current_table 2>&1 | tee -a $EXPORT_DIR/hbase-export.log | grep mapreduce.Job
    echo "$CONTROLPLANE_TENANT_ID-$current_table export finished"
done
END=$(date +%s)
echo $((END-START)) | awk '{print "Export time execution -> "int($1/60)" min "int($1%60)" sec"}'

    3. Exit the HBase container.

      exit

  4. As the root user on the Control Center master, backup existing HBase folders.

    TENANT_ID=$(serviced service status Zenoss.resmgr | sed -n '2p' | awk {'print $2'})
cd /opt/serviced/var/volumes/$TENANT_ID
tar -czf hbase-master.tar.gz hbase-master/
tar -czf hbase-zk1.tar.gz hbase-zookeeper-1/
tar -czf hbase-zk2.tar.gz hbase-zookeeper-2/
tar -czf hbase-zk3.tar.gz hbase-zookeeper-3/

  5. As the root user on the Control Center master, stop the HBase cluster and remove all HBase data, except the table export from Step 3.

    1. Stop the HBase service.

      serviced service stop Hbase

      To confirm it has stopped, you may use:

      serviced service status Zenoss.resmgr

    2. Remove the HBase data, except the table export.

      TENANT_ID=$(serviced service status Zenoss.resmgr | sed -n '2p' | awk {'print $2'})
cd /opt/serviced/var/volumes/$TENANT_ID
rm -rf hbase-zookeeper-1/* hbase-zookeeper-2/* hbase-zookeeper-3/*
rm -rf hbase-master/!(hdfsTableExport)

  6. Roll the HBase and OpenTSDB services forward to use the new Docker images included in Resource Manager 6.8.0.

    TENANT_ID=$(serviced service status Zenoss.resmgr | sed -n '2p' | awk {'print $2'})
serviced docker override localhost:5000/$TENANT_ID/hbase:latest zenoss/hbase:24.0.10
serviced docker override localhost:5000/$TENANT_ID/opentsdb:latest zenoss/hbase:24.0.10
serviced docker sync

  7. Perform the HBase data migration.

    1. Start the HBase cluster.

      serviced service start Hbase

    2. Attach to the HBase container

      serviced service attach hmaster su - hbase

    3. Create new tables for the HBase cluster.

      /opt/opentsdb/create_table_splits.rb tsdb t 
/opt/opentsdb/create_table_splits.rb $CONTROLPLANE_TENANT_ID-tsdb t
HBASE_HOME=/opt/hbase COMPRESSION=SNAPPY /opt/opentsdb/create_table_splits.sh
HBASE_HOME=/opt/hbase COMPRESSION=SNAPPY TSDB_TABLE=$CONTROLPLANE_TENANT_ID-tsdb UID_TABLE=$CONTROLPLANE_TENANT_ID-tsdb-uid TREE_TABLE=$CONTROLPLANE_TENANT_ID-tsdb-tree META_TABLE=$CONTROLPLANE_TENANT_ID-tsdb-meta /opt/opentsdb/create_table_splits.sh

    4. Import the exported data from the previous HBase cluster version.

      HBASE_HOME=/opt/hbase
EXPORT_DIR=/var/hbase/hdfsTableExport
MAPRED_JAVA_PATH=org.apache.hadoop.hbase.mapreduce
START=$(date +%s)
for current_table in tsdb tsdb-meta tsdb-tree tsdb-uid
do
    echo "$current_table import started"
    $HBASE_HOME/bin/hbase $MAPRED_JAVA_PATH.Import $current_table $EXPORT_DIR/$current_table 2>&1 | tee -a $EXPORT_DIR/hbase-import.log | grep mapreduce.Job
    echo "$current_table import finished"
    echo "$CONTROLPLANE_TENANT_ID-$current_table import started"
    $HBASE_HOME/bin/hbase $MAPRED_JAVA_PATH.Import $CONTROLPLANE_TENANT_ID-$current_table $EXPORT_DIR/$CONTROLPLANE_TENANT_ID-$current_table 2>&1 | tee -a $EXPORT_DIR/hbase-import.log | grep mapreduce.Job
    echo "$CONTROLPLANE_TENANT_ID-$current_table import finished"
done
END=$(date +%s)
echo $((END-START)) | awk '{print "Import time execution -> "int($1/60)" min "int($1%60)" sec"}'

  8. Start the remaining Resource Manager services.

    serviced service start Zenoss.resmgr

Perform post-update tasks

Remove the pre-upgrade snapshot and start Resource Manager

The Resource Manager update script uses Control Center to create and tag a snapshot of the system before it begins the update process. Tagged snapshots persist until they are explicitly removed, and grow over time. When you are satisfied the new release is working properly, remove the pre-update snapshot.

  1. Log in to the Control Center master host as a user with serviced CLI privileges.

  2. Display a list of all Control Center snapshots, with their tags.

    serviced snapshot list -t

    Example result, truncated to save space:

    Snapshot                         Description  Tags
xm5mtezbyo2_20160211-220535.480               preupgrade-resmgr-6.6.0

    The snapshot identifier is shown in the first column.

  3. Remove the pre-update snapshot.

    Replace Snapshot-ID with the identifier of the pre-update snapshot returned in the previous step:

    serviced snapshot remove Snapshot-ID

  4. Start Resource Manager.

    serviced service start Zenoss.resmgr

Optional: Install the application template

The most recent Resource Manager application template is not included in non-appliance updates and is not needed. Use the following procedure if you plan to delete a deployed application template and then redeploy a newer version of the template. (For example, in a development or staging environment.)

Follow these steps

  1. In a web browser, navigate to the delivery.zenoss.io site, and then log in.

  2. Download the template package file.

    zenoss-resmgr-service-6.8.0-1.noarch.rpm

  3. Copy the package file to the Control Center master host.

  4. Log in to the Control Center master host as root or as a user with superuser privileges.

  5. Change directory to the directory where the template package file is located.

  6. Install the template.

    yum install ./zenoss-resmgr-service-6.8.0-1.noarch.rpm

    The template file is stored in /opt/serviced/templates.