Resource Manager 6.3.2

This section contains important information about release 6.3.2 of Zenoss Resource Manager (Resource Manager).

New features

Zenhub scaling enhancements

(ZEN-30832, originally identified as defect ZEN-29812). Previously the zenhub service delegated work to subprocesses, which limited its scale to the resources available on a single host. Now, the zenhub service can delegate (non-invalidation related) work to a separate service, zenhubworker, and you can run as many zenhubworker services as required, on any host in the resource pool. The upgrade process will automatically reconfigure zenhub appropriately for horizontally scaling. However. if you have previously had a "superhub" configuration (characterized by having single zenhub instance in a dedicated Control Center resource pool) put in place by Zenoss Services or Support to help mitigate scaling issues this enhancement now addresses, you may also wish to adjust its resource pool configuration manually, after the upgrade. For more information about updating "superhub" configurations, see Migrating zenhub service pool configuration.

Zope deadlock improvements

This release, along with the Control Center 1.6.3 release, provide the following enhancements to Zope-based services:

(CC-3559) Automatic restart when a service fails 3 health checks in a row. This feature ensures deadlocked services are restarted after 3 failed health checks (2.5 minutes). The zauth, zenreports, zope, and zenapi services benefit from this update.
(ZEN-30871) The default number of zauth service instances is now two, so that services which depend on authentication are not blocked when zauth is deadlocked.
(ZEN-29781) A supporting library was upgraded to avoid a thread lock issue in a previous version.

Debug option for device modeling

(ZEN-30344) The browser interface now includes an option to model a device with debug messages enabled.

Configurable query limit

(ZEN-29877) The Solr index service now returns a default limit of 10,000 results. This can be changed by editing global.conf, if desired. The variable name is global.conf.solr-search-limit.

Default "HostPolicy" for several RM services has been changed to PREFER_SEPARATE

(ZEN-22178) The default host policy for zenhub, zeneventd, and zope services has been change from the default of "none" to PREFER_SEPERATE. The default policy for the new "zenhubworker" service added by ZEN-30832, has also been set to PREFER_SEPARATE. This HostPolicy setting has no effect for zenhub since it instance maximum was set by ZEN-30832 to one, but has been set this way explicitly for future proofing in case that maximum is raised in the future. This change is also automatically applied by upgrades if the HostPolicy is not already set to PREFER_SEPARATE.

Considerations and workarounds

Compatibility with Zenoss Service Impact

This version of Resource Manager is compatible with Zenoss Service Impact version 5.2.3 or later.

Regular expression matching limitation

(ZEN-29376) Resource Manager uses the Apache Solr search platform, which tokenizes fields. Currently, searching a tokenized field with a regex matches only a single token, limiting complex regex searches.

For example, on the INFRASTRUCTURE page, IP Address is a tokenized field. Regex search ^192.168 will not work to filter only IP addresses that start with 192.168. The Device column is untokenized; regex search on the device name works as expected.

Load time for component graphs

(ZEN-29300) Load time might exceed 10 seconds for more than 200 component graphs when you have activated the All on same graph check box.

NFS client 4.1 is not supported

A file locking defect might exist in NFS 4.1 with RHEL/CentOS 7.4, which could cause zeneventserver to crash and other DFS-related problems. For more information, see Configuring NFS 4.0 or knowledge base article Potential Issues Running With RHEL 7.4 Or CentOS 7.4.

CentralQuery maximum memory allocation pool

(ZEN-15907) The size of the CentralQuery maximum memory allocation pool is set by the RAMCommitment variable in the CentralQuery service definition. The default value is 1024MB. Do not change the value to anything less than 1024MB.

High-availability documentation

Customers who have deployed Control Center and Resource Manager in a high-availability configuration are still supported in this release. For upgrade instructions, refer to the Control Center Upgrade Guide for High-availability Deployments.

Upgrade considerations

(ZEN-29807) Upgrading from Resource Manager 5.x to 6.x includes a required step to reindex all cataloged objects into the new Solr index. To speed this process, before upgrading, you can increase the CPU core count to use more workers. For more information, see knowledge base article Reducing 5.X To 6.X Upgrade Times By Increasing Workers For Indexing.
Beginning with release 6.2.1, support for upgrading versions earlier than 5.3.3 was removed from the upgrade instructions. For assistance with planning for upgrading versions prior to 5.3.3, please contact Zenoss Support. In short, if you are on a version prior to that, we'll recommend you should plan to upgrade to version 5.3.3 and then to the latest 6.x version from there.
The upgrade process might reinstall an older version of the Catalog Service ZenPack. To avoid this issue, manually remove older versions of the Catalog Service ZenPack .egg file from the ZenPack backups directory.
(ZEN-28375) Beginning with Resource Manager 6.1.0, for counter/derive (rate) data points, the calculated rate value is stored instead of raw counters. The rate is calculated at the collector daemon as it collects data. If a graph (or API) request for a derive of counter data point spans the upgrade data, the system automatically queries the data correctly. However, slight anomalies might occur in the data that is captured immediately before and after an upgrade from Resource Manager 6.0.1 or earlier.
(ZEN-29100) No data is collected for the ZenossRM device after an upgrade if the localhost collector is moved off the master host. If you have moved the localhost collector off the Control Center master host, to enable data collection, set properties for the ZenossRM device as follows:
1. Navigate to the ZenossRM device overview page and select Configuration Properties.
2. Set the following zProperties:
  
  zRMMonCCHost - Enter the IP address of the Control Center master host. zRMMonCCUser - Enter the name of the ccuser account, which is the default account for gaining access to the Control Center browser interface. zRMMonCCPassword - Enter the password of the ccuser account.

Migrating zenhub service pool configuration

As a result of enhancement ZEN-30832, the zenhub service can now scale horizontally; that is, zenhub is no longer limited by the resources available to a single Control Center delegate. Your current configuration will be automatically migrate to take advantage of this capability.

If you have previously had a "superhub" configuration (characterized by having single zenhub instance in a dedicated Control Center resource pool) put in place by Zenoss Services or Support to help mitigate scaling issues this enhancement now addresses, you should plan to manually adjust the pool this service is in. In short, there is no longer a need to a separate pool, and you should follow the follow manual process post upgrade to revert such configuration to make best use of deploy resources.

In the Control Center browser interface, edit the "zenhub" service and note the pool is it configured for. Typically this is something like "zenhub".
In the Control Center browser interface, note the pool the top level service is configured for (edit service right at the top of the Zenoss application page". Typically this is something like "default" or "resource manager"
Determine whether the pool in step 2 is currently properly resourced to accept the load from zenhub. Check the pool as a whole can support the increased typical memory usage for zenhub, and that the pool as a whole can support the additional CPU load for zenhub (approximately 1+(number of zenhubworker instances)+(0.5 * number of invalidation workers listed in zenhub.conf) cores of work).
Stop the zenhub and zenhubworker services.
In the Control Center browser interface, edit the zenhub service and change the pool configuration from the pool noted in step 1, to the pool noted in step 2.
Repeat step 5 for the zenhubworker service.
If you need to move the existing delegate from the pool in step 1 to the pool in step 2, based on the results of the analysis in step 3, do so now.
Start the zenhub and zenhubworker services.
If you need did NOT need to move the existing delegate from the pool in step 1 to the pool in step 2, you can deregister the delegate from the pool now.
Delete the pool identified in step 1.

Fixed issues

ID	Description
ZEN-31101	The zenhub service is performing tasks that should be performed by the zenhubworker service.
ZEN-22178	Default "HostPolicy" for several RM services has been changed to PREFER_SEPARATE
ZEN-30966	On the INFRASTRUCTURE page, an error occurs when filtering devices by priority.
ZEN-30945	When making a local copy of a template, the original template is displayed.
ZEN-30892	Modeling jobs initiated in the browser interface are not given priority and may be delayed.
ZEN-30871	The default number of Zauth instance is too low. See Zope deadlock improvements.
ZEN-30866	No debug data is written to the ZODB `connection_info` table.
ZEN-30835	SNMP discovery is not parallel.
ZEN-30832 ZEN-29857	The zenhub service cannot scale horizontally. See Zenhub scaling enhancements.
ZEN-30803	The LDAP plugin does not support adding trusted certificates.
ZEN-30795	SNMP traps sent through a NAT do not include source IP address.
ZEN-30790	Devices added with the default device class (/) can't be found by class.
ZEN-30789	Zenbatchload displays manageIp warning on all devices.
ZEN-30774	The Ping Status Issues and SNMP Status Issues reports don't work on new installations.
ZEN-30695	The modeler plugins for the /Server/SSH/Linux class are SNMP, not command.
ZEN-30579	The defaults of the Add maintenance window dialog box do not match its most frequent use case.
ZEN-30540	Reports run automatically when selected.
ZEN-30438	Modeling and monitoring with SNMP v1 does not work properly.
ZEN-30437	API method allows non-privileged users to view privileged data.
ZEN-30388	Unable to lock Dashboard portlets from updates.
ZEN-30363 ZEN-29514	The version of OpenSSL included in Resource Manager (1.0.1) is no longer supported.
ZEN-30361	The warning messages from the zenmib utility are not readable (one character per line).
ZEN-30344	Unable to view debug information about a modeling attempt from the browser interface. See Debug option for device modeling.
ZEN-30325	Moving devices from /Discovered to another class fails, leaving devices in an inconsistent state.
ZEN-30024	The zenmib utility is unable to process overlapping ranges.
ZEN-29982	The reportmail feature does not work for graph or multi-graph reports.
ZEN-29978	AWS account password shown in plain text on Firefox browser.
ZEN-29929	The password field in ServiceNow notifications loses the entries in the password fields when other fields are subsequently filled.
ZEN-29927 ZEN-29898	The zenperfsnmp daemon does not close sockets after collections and eventually stops working.
ZEN-29878	The Solr service logs too many INFO messages.
ZEN-29877	The Solr service does not limit the number of results that can be returned for a query. See Configurable query limit.
ZEN-29856	The zencatalogscan utility times out and eventually fails.
ZEN-29838	In browser interfaces, flares that should include hyperlinks have only HTML elements.
ZEN-29796	The Edit Trigger dialog does not resize properly.
ZEN-29781	Zope services can become deadlocked. See Zope deadlock improvements.
ZEN-29775	The Zope self-monitoring script hangs when the instance is unresponsive but still listening on TCP. See Zope deadlock improvements.
ZEN-29764	Default configuration of maximum connections for MariaDB does not match recommended configuration.
ZEN-29762	The default minimum RAM requirement for the `memcached` service is too small.
ZEN-29702	Upgrade script reads the FROM version from the wrong location after an initial upgrade failure.
ZEN-29654	The zenmail service is unable to parse the hostname from an email address.
ZEN-29646	Silent failure when re-identifying a device that has no performance.
ZEN-29553	When the SSH password is incorrect, monitoring fails and the resulting event flaps (auto-clear and re-appear).
ZEN-29542	Asynchronous jobs for moving devices fail with `global name 'log' is not defined` message.
ZEN-29469	(MultiRealm ZenPack) The IP addresses associated with a device do not move to a new realm until the device is remodeled.
ZEN-29452	When adding multiple devices, the duration field starts at a large negative number instead of zero.
ZEN-29430	Monitoring template name handling does not support case-sensitive names.
ZEN-29393	The default minimum RAM requirement for the `zenhub` service is too small.
ZEN-29308	The upgrade log does not include a summary of the migration scripts that were run during an upgrade.
ZEN-29264	Large multi-graph reports can't open in new tab due to long URLs.
ZEN-29220	MariaDB does not perform well with the default memory allocator `glibc`; switch to `jemalloc`.
ZEN-29062	Changes made to the LDAP user object classes value outside of the browser interface are not maintained when the LDAP properties form is saved in the browser interface.
ZEN-28887	No option to disable the removal of HTML/XML tags from plaintext email notifications.
ZEN-28676	The page notification type is unavailable, so integrations with external paging systems is not possible.
ZEN-28619	Time values are used and displayed inconsistently in the browser interface.
ZEN-27106	The `zeneventserver` log can get flooded with messages about events being dropped due to missing fields.
ZEN-25930	Unable to specify fields to include/exclude when exporting events.

Known issues

The following list describes the known issues in this release:

(ZPS-4986,ZenPacks.zenoss.HttpMonitor, v3.0.4) PythonCollector crashes when the IP Address or Proxy Address field is blank. After the crash, the zenpython debug log includes content similar to the following example:
```
2018-12-14 17:31:09,797 INFO zen.zenpython.twisted: Stopping protocol <twisted.names.dns.DNSDatagramProtocol object at 0x94599fd0>
2018-12-14 17:31:09,798 INFO zen.zenpython.twisted: DNSDatagramProtocol starting on 57548
```
To avoid the crashes, provide content for the appropriate field(s).
(ZPS-4473,ZenPacks.zenoss.Microsoft.Windows, v2.9.1) When a Windows device reboots, data from PerfMon data sources do not resume collection. The workaround is to remodel the device.
(ZEN-31194) This release includes ZenPacks.zenoss.LDAPAuthenticator v3.3.3, which no longer includes an option to skip SSL certificate verification during installation or upgrade. If you are using a self-signed certificate, upload it before upgrading. For more information, see Adding an SSL certificate.
(ZEN-30457) When processing messages from large devices, zenhub invalidation workers can get disconnected from Rabbit MQ due to heartbeat timeouts. If you are seeing the invalidations queue grow without also coming back down, or the number of consumers on the invalidations queue less than the number of running workers, please contact Zenoss Support.
This release includes an update of the Python urllib3 library, from version 1.10.2 to version 1.22. The new version checks TLS/SSL certificates by default. If you are using a self-signed certificate, upload it before updating Resource Manager or ZenPacks that rely on urllib3. This change also affects Zenoss Analytics (ZEN-31334). For more information, refer to the Zenoss Support Knowledgebase.

Installed ZenPacks

This section lists ZenPacks that are automatically installed, those that are packaged but not installed, and those that are obsolete and should be uninstalled. For more information about ZenPacks, see the ZenPack catalog.

This release of Resource Manager installs the following ZenPacks at the current version listed in the table.

ZenPack	Current version	Previous version
ZenPacks.zenoss.AdvancedSearch	2.0.1	2.0.0
ZenPacks.zenoss.AixMonitor	2.2.3	Same
ZenPacks.zenoss.ApacheMonitor	2.1.4	Same
ZenPacks.zenoss.AuditLog	1.4.1	Same
ZenPacks.zenoss.AWS	4.0.2	4.0.1
ZenPacks.zenoss.CalculatedPerformance	2.5.1	2.5.0
ZenPacks.zenoss.CiscoMonitor	5.9.0	Same
ZenPacks.zenoss.CiscoUCS	2.8.0	2.7.0
ZenPacks.zenoss.ComponentGroups	1.7.0	1.6.0
ZenPacks.zenoss.ControlCenter	1.6.2	Same
ZenPacks.zenoss.Dashboard	1.3.3	1.2.9
ZenPacks.zenoss.Dell.PowerEdge	2.0.4	Same
ZenPacks.zenoss.Diagram	1.3.2	1.3.1
ZenPacks.zenoss.DistributedCollector	3.1.7	3.1.6
ZenPacks.zenoss.DnsMonitor	3.0.1	Same
ZenPacks.zenoss.Docker	2.0.4	2.0.3
ZenPacks.zenoss.DurationThreshold	2.0.5	Same
ZenPacks.zenoss.DynamicView	1.6.2	1.6.1
ZenPacks.zenoss.EMC.base	2.1.0	Same
ZenPacks.zenoss.EnterpriseCollector	1.8.3	Same
ZenPacks.zenoss.EnterpriseReports	2.5.0	Same
ZenPacks.zenoss.EnterpriseSecurity	1.2.0	Same
ZenPacks.zenoss.EnterpriseSkin	3.3.5	Same
ZenPacks.zenoss.HP.Proliant	3.3.2	Same
ZenPacks.zenoss.HttpMonitor	3.0.4	Same
ZenPacks.zenoss.IBM.Power	1.1.2	Same
ZenPacks.zenoss.InstalledTemplatesReport	1.1.1	Same
ZenPacks.zenoss.JuniperMonitor	2.1.1	Same
ZenPacks.zenoss.LDAPAuthenticator	3.3.3	3.3.1
ZenPacks.zenoss.LDAPMonitor	1.4.2	Same
ZenPacks.zenoss.Licensing	0.3.0	0.2.1
ZenPacks.zenoss.LinuxMonitor	2.3.2	2.2.7
ZenPacks.zenoss.Microsoft.Azure	1.3.3	1.3.0
ZenPacks.zenoss.Microsoft.Windows	2.9.2	2.9.0
ZenPacks.zenoss.MySqlMonitor	3.1.0	Same
ZenPacks.zenoss.NetAppMonitor	3.6.0	Same
ZenPacks.zenoss.NtpMonitor	3.0.0	Same
ZenPacks.zenoss.PredictiveThreshold	1.3.0	1.2.2
ZenPacks.zenoss.PropertyMonitor	1.1.1	Same
ZenPacks.zenoss.PythonCollector	1.10.1	Same
ZenPacks.zenoss.RMMonitor	1.1.1	1.1.0
ZenPacks.zenoss.SolarisMonitor	2.5.1	Same
ZenPacks.zenoss.StorageBase	1.4.3	Same
ZenPacks.zenoss.SupportBundle	1.1.2	Same
ZenPacks.zenoss.vSphere	4.0.0	3.7.2
ZenPacks.zenoss.WBEM	2.1.0	2.0.1
ZenPacks.zenoss.WSMAN	1.0.3	1.0.1
ZenPacks.zenoss.ZenDeviceACL	2.3.0	Same
ZenPacks.zenoss.ZenJMX	3.12.1	Same
ZenPacks.zenoss.ZenMail	5.1.0	Same
ZenPacks.zenoss.ZenOperatorRole	2.2.0	Same
ZenPacks.zenoss.ZenPackLib	2.1.1	Same
ZenPacks.zenoss.ZenSQLTx	2.7.1	Same
ZenPacks.zenoss.ZenWebTx	3.0.3	Same

Packaged ZenPacks

The following ZenPacks are packaged with Resource Manager, but not automatically installed:

ZenPacks.zenoss.BigIpMonitor
ZenPacks.zenoss.BrocadeMonitor
ZenPacks.zenoss.Ceph
ZenPacks.zenoss.CheckPointMonitor
ZenPacks.zenoss.CiscoAPIC
ZenPacks.zenoss.DatabaseMonitor
ZenPacks.zenoss.DB2
ZenPacks.zenoss.HpuxMonitor
ZenPacks.zenoss.JBossMonitor
ZenPacks.zenoss.Memcached
ZenPacks.zenoss.Microsoft.Exchange
ZenPacks.zenoss.Microsoft.Lync
ZenPacks.zenoss.Microsoft.MSMQ
ZenPacks.zenoss.NetScaler
ZenPacks.zenoss.NetScreenMonitor
ZenPacks.zenoss.NSX
ZenPacks.zenoss.OpenStack
ZenPacks.zenoss.OpenStackInfrastructure
ZenPacks.zenoss.OpenvSwitch
ZenPacks.zenoss.PostgreSQL
ZenPacks.zenoss.RabbitMQ
ZenPacks.zenoss.TomcatMonitor
ZenPacks.zenoss.XenServer

Obsolete ZenPacks

ZenPacks.zenoss.ZenMailTX is obsolete. If this ZenPack is installed at your site, uninstall it.

Zenoss JSON API changes

The Zenoss JSON API includes the following changes from version 6.2.0:

(ZEN-29558) The getDevices method of ModelQuery router is updated to support offsets. The method requires additional fields, and the default limit on the number of devices it returns is 200.
The exportDevices and importDevices methods of the DeviceDumpLoadRouter class now require ZenManager privileges.
The DeviceRouter class includes the following enhancements:
- The getInfo method now includes improved security for password fields.
- The setCollector method includes improved handling of debug messages.
- The addDevice method includes a bug fix in the duplicate device message.