Installing Analytics

Before installing Analytics, review the release notes.

Note

The instructions in this section are written for CentOS/RHEL 7.2+, which are the officially supported and tested versions. However, field evidence indicates that Analytics functions without issue on CentOS/RHEL 5 and 6. If you cannot deploy CentOS/RHEL 7.2+, consult Zenoss Professional Services for assistance. (Using CentOS/RHEL 5 or 6 might not prevent a successful installation; however, it is important to plan to move your Analytics installation to CentOS/RHEL 7.2+.)

Preparing to install

Analytics data is stored in a mySQL database. Like any database server, compute speed, memory usage, and disk speed are critical for the Analytics server. The installation is performed on a single server that must have adequate CPU, memory, and disk resources. For Zenoss Resource Manager customers, your Zenoss professional services architect will provide specific sizing for your current needs and that allows for anticipated growth of the data warehouse.

In all scenarios, Zenoss recommends provisioning the Analytics server as a virtual machine (VM) to allow flexible rescaling of CPU and memory resources on an ongoing basis. Provision storage to be extensible also. Best results are achieved when the Analytics server is a stand-alone server that is dedicated to the installation of Analytics.

Other than the operating system (OS) and the external dependencies explicitly listed by this document, run no third-party software on the server. In particular, anti-virus software is known to significantly interfere with the speed of all database systems, and Analytics is no exception. Before starting the installation procedure, audit your OS environment to check for and disable third-party software.

Verifying candidate host resources

The minimum requirements for resources are as follows:

CentOS/RHEL 7.2+
50GB of available storage at /opt/zenoss_analytics
500GB of available storage at /var/lib/mysql
10GB of available storage at /tmp
8GB RAM
8 CPU cores

Validating the OS environment

Log in to the candidate host as root or as a user with superuser privileges.
Verify that the host implements the 64-bit version of the x86 instruction set:
```
uname -m
```
- If the output is x86_64, the architecture is 64-bit. Proceed to the next step.
- If the output is i386/i486/i586/i686, the architecture is 32-bit. Stop this procedure and select a different host for the Analytics server.
Verify that name resolution works on this host:
```
hostname -i
```
- If the result is not a valid IPv4 address, add an entry for the host to the network nameserver, or to /etc/hosts.
- Ensure that the hostname also resolves to the loopback interface locally by editing /etc/hosts to include an entry for the hostname of the server after the listing for 127.0.0.1.
Update the operating system to the latest CentOS/RHEL version (currently 7.5), if desired.
1. Determine which release is installed:
```
cat /etc/redhat-release
```
2. If you are not running the latest version (currently 7.5), before starting the Analytics installation process, update the operating system if desired and then reboot the server.
```
yum update -y && reboot
```
  Note: There is an issue with file locking when using NFS 4.1 with CentOS/RHEL 7.4 specifically (was fixed in 7.5). Updating the OS to version 7.4 requires that you force the use of NFS 4.0 to avoid this issue. For detailed steps, refer to the following Zenoss knowledgebase article: Potential Issues Running With RHEL 7.4 Or CentOS 7.4.
Verify that UID 1337 is available to Analytics:
```
getent passwd 1337
```
Determine whether the firewalld service is enabled, and if necessary, disable the firewall.
```
systemctl status firewalld.service
```
- If the result includes Active: inactive (dead), the service is correctly disabled. Proceed with the next step.
- If the result includes Active: active (running), the service is enabled and must be disabled now and on the next boot:
```
systemctl stop firewalld && systemctl disable firewalld
```
Determine whether Security-Enhanced Linux (SELinux) is installed and enabled, and if so, disable it:
```
test -f /etc/selinux/config && grep '^SELINUX=' /etc/selinux/config
```
If the command does not return a result, proceed with the next step. If the command returns a result, SELinux is installed. Disable it and reboot the server:
1. Open /etc/selinux/config in a text editor and change the value of the SELINUX variable to disabled.
2. Confirm the new setting:
```
grep '^SELINUX=' /etc/selinux/config
```
3. Reboot the server:
```
reboot
```
Install and configure the NTP package:
```
ntpd -gq
```
Enable the ntpd daemon:
```
systemctl enable ntpd
```

Checking memory and CPU

Determine whether the available memory and swap is sufficient.

The Analytics server requires a minimum of 8GB of available memory and a minimum of 8 cores.

Log in to the candidate host as root or as a user with superuser privileges.
Display the available memory:
```
free -h
```
Compare the available memory with the amount required for the Analytics server.
Execute top, then press 1. Note the number of CPU cores present.

Provisioning storage

Determine whether the available, unused storage is sufficient and provision it. The required minimums are as follows:

50GB for /opt/zenoss_analytics. This is where the Analytics software is installed, and is also used as temporary storage by the ETL processes.
500GB for /var/lib/mysql. This is where mySQL stores its data for both the data Analytics database and the data warehouse.
10GB for /tmp. This is used as temporary storage by both the Analytics ETL application and mySQL.
Log in to the candidate host as root or as a user with superuser privileges.
Display the available storage devices:
```
lsblk --output=NAME,SIZE
```
Compare the available storage with the amount required for an Analytics server, identify an appropriate partition, and then create the appropriate file system.
For each type of storage that requires provisioning, identify an appropriate partition, and create and mount the appropriate file system.
1. Identify the target primary partition for the file system to create.
```
lsblk --output=NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT
```
2. Create an EXT4 file system of the partition.
```
mkfs -t ext4 <partition>
```
3. Create the mount point for the file system.
```
mkdir -p <filesystem path>
```
  For example, mkdir -p /opt/zenoss_analytics.
4. Add an entry for the file system to the /etc/fstab file. Replace <partition> with the path of the partition from step b and the <filesystem path> from that used in the previous step:
```
echo "<partition> <filesystem path> ext4 defaults 0 0" >> /etc/fstab
```
5. Mount the file system, and then verify that it mounted correctly. Replace the <filesystem path> with the actual path.
```
mount -a && mount | grep <filesystem path>
```

Opening network ports

All Resource Manager 4.x collector hosts, a Resource Manager 4.x master node and all Resource Manager 5.x and 6.x hosts (including collector pool hosts) must be able to access the Analytics server HTTPS address. Zenoss strongly recommends that, before installing Analytics, you add the FQDN of the Analytics server to your DNS.

The standard SSL port of 443 is used for all Analytics -related communication between Resource Manager hosts and the Analytics server.

Optionally, to enable the use of third-party tools with Analytics, the following TCP ports can be opened on the Analytics server:

mySQL server port, 3306, which allows direct mySQL access to the data warehouse and Jaspersoft databases.
jasperserver port, 7070, which allows direct connection from Jaspersoft development tools to the Jaspersoft installation.

Installing the OpenJDK

Analytics requires the OpenJDK 7 Development package. Analytics cannot be installed when just the JRE is installed.

Prior to installing the OpenJDK 7 Development package, you should remove all other versions of Java that are on the system:

yum -y remove $(rpm -qa | grep -Ei '(jdk|jre|java)')

Log in to the Analytics server as root, or as a user with superuser privileges.

Install the OpenJDK 7.

yum -y install java-1.7.0-openjdk-devel

Verify that the installation succeeded.
```
java -version
```
If the command returns output similar to the following, continue to the next procedure. The <_version> patch version does not need to match, the latest version of 1.7.0 is fine. 1.8.0+ is NOT supported.
```
java version "1.7.0_75"
OpenJDK Runtime Environment (rhel-2.5.4.7.el7_1.x86_64 u75-b13)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
```

Setting the JAVA_HOME environment variable

After installing the OpenJDK, you must set $JAVA_HOME for the root user.

Log in to the Analytics server as root, or as a user with superuser privileges and check that the variable is not currently set:
```
echo $JAVA_HOME
```
Locate and note the path to your OpenJDK directory. Typically, this is /usr/lib/jvm/java-1.7.0-openjdk-<version>.
```
ls -l /usr/lib/jvm/java-1.7.0-openjdk-<version>
```
Navigate to the /etc/default directory.
```
cd /etc/default
```
Open a new zenoss_analytics file in an editor.
```
vi zenoss_analytics
```
Edit the JAVA_HOME line in zenoss_analytics to point to your OpenJDK location and save the file. The resulting entry will look like this:
```
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-<version>
```
Source this file:
```
source zenoss_analytics
```
Test the new setting of the JAVA_HOME variable:
```
echo $JAVA_HOME
```

Installing MariaDB

MariaDB is used as the mySQL database software for Analytics databases. Version 10.0.x is the currently supported version.

Version 10.1+ is known NOT to work, and is NOT supported.
Amazon RDS is known NOT to work, and is NOT supported. Specifically RDS does not support binary UDFs, which means that nth-percentile aggregations will not work; there may be other issues that we are not aware of.

Make sure there is no existing installation of mysql server in place. As the root user, execute the following:

rpm -qa | grep -i mysql-server

If one exists, remove it before proceeding.

If it isn't already installed, install the Perl Database Interface Module (DBI) with the following command as the root user:

yum -y install perl-DBI

Create a custom MariaDB version 10.0.x YUM repository. The following example shows information for a CentOS 7 installation. For more information, see the MariaDB repository.
1. As the root user, navigate to the /etc/yum.repos.d directory.
```
cd /etc/yum.repos.d
```
2. Edit a file with vi called MariaDB.repo and cut and paste the following text into the file.
```
# MariaDB 10.0 CentOS repository list
#http://mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.0/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
```
When the MariaDB.repo file is in place, install the MariaDB server:
```
yum -y install mariadb-server
```
Note: If you have not accepted the MariaDB GPG key, you will be prompted to do so.

Add the following configuration settings:

echo "[mysqld]" >> /etc/my.cnf
echo "table_open_cache=16K" >> /etc/my.cnf
echo "table_definition_cache=16K" >> /etc/my.cnf
echo "tmp_table_size=2G" >> /etc/my.cnf
echo "max_heap_table_size=2G" >> /etc/my.cnf
echo "join_buffer_size=512K" >> /etc/my.cnf
echo "open_files_limit=200000" >> /etc/my.cnf
echo "tmpdir=/tmp" >> /etc/my.cnf
echo "wait_timeout=86400" >> /etc/my.cnf
echo "innodb_adaptive_hash_index=OFF" >> /etc/my.cnf
echo "innodb_buffer_pool_size=6G" >> /etc/my.cnf
echo "innodb_log_file_size=1892M" >> /etc/my.cnf
echo "innodb_log_buffer_size=128M" >> /etc/my.cnf
echo "optimizer_search_depth=0" >> /etc/my.cnf

Note: The values for the innodb settings should reflect your system. Edit the file in vi and change the default innodb_buffer_pool_size of 16G to 50% of the total memory of the server and set innodb_log_file_size default size of 1892M to 25% of the innodb_buffer_pool_size value.

Start the MariaDB mysql server for the first time:
```
/etc/init.d/mysql start
```
You can confirm success by connecting to the database server as follows:
```
mysql -u root
```
If mysql fails to start, it is highly likely you have a /etc/my.cnf configuration file misconfiguration or environmental preparation was not completed correctly. You can check the /var/lib/mysql/<serverfqdn>.err file for details of why startup was not successful. You should correct any issues and confirm successful server start before proceeding further with the installation.
Set MariaDB to start on boot:
```
systemctl enable mysql
```
Optionally, if you wish to enable remote access to the database server (that is, you have opened port 3306), you can grant remote access to the root and reporting_read users (Analytics installation will create the later as a read-only user otherwise.)
```
mysql -u root
grant all privileges on *.* to 'root'@'%' with grant option;
grant all privileges on *.* to 'reporting'@'%' with grant option;
flush privileges;
```

Installing the Analytics server

Install the Analytics server by using an .rpm file on a separate server.

To install the Analytics ETL and embedded Jaspersoft software, enter the following command, as the root user, based on the RPM version that you are installing:
```
rpm -ivh zenoss_analytics-<version>.noarch.rpm
```
Replace <version> with the version number you want to install; for example, 5.1.0-1.
Change to the zenoss user and execute the initial database creation procedure. This procedure creates the jaspersoft database (zenoss_analytics), populates it with Zenoss examples and creates a blank data warehouse database (reporting):
```
su - zenoss
/opt/zenoss_analytics/bin/upgrade_db.py
```
If this process ends in errors, an environmental issue exists. Typically, the environmental procedure was not followed. In the first instance, make sure localhost resolves to 127.0.0.1 and that you do not have files in /tmp left over from a previous installation attempt. Correct issues and then rerun the above process as the zenoss user.
After this process completes, change to the root user and enable Analytics to start on boot and then start the service.
```
exit
systemctl enable zenoss_analytics
systemctl start zenoss_analytics
```
Tail the Analytics application log (this is a Tomcat log) and watch the log as Analytics starts up for the first time:
```
tail -F /opt/zenoss_analytics/logs/catalina.out
```
On first startup, the data warehouse schema is adjusted and populated with configuration information that is appropriate to the current version of Analytics. This process usually takes about 5 minutes, during which the Analytics application is unavailable. Wait for the following message in this log and then continue with the next step.
```
INFO: Server successfully started in <time in ms>
```
If this first startup ends in errors, an environmental issue exists. Typically, the environmental procedure was not followed. In the first instance, make sure you have the correct version of mariadb installed. Version 10.0.x is required.
```
rpm -qa | grep -i maria
```

Installing the percentile function

The mysql SQL windowing functions are rather limited. To enable a sql database function for calculating percentiles (used by the Analytics percentile aggregation capabilities), Zenoss has written a mysql User Defined Function (UDF) and this needs to be installed into mysql. To install this function, log in to the Analytics server as root, or as a user with superuser privileges and execute the following procedure to install this function and restart the sql server (required by installation of any UDF):

Log in as the root user or a user with superuser permissions and execute the following commands:

su - zenoss
/opt/zenoss_analytics/bin/setup_zenoss_extensions install
exit
service zenoss_analytics stop
/etc/init.d/mysql restart
service zenoss_analytics start

Securing Analytics to use SSL

Analytics uses Apache and mod_ssl to provide SSL for all of the different types of communication needed. It is assumed that if you are running Resource Manager 4.x you already have secured the Resource Manager behind SSL (this is in place as a matter of course in Resource Manager 5.x and 6.x). This is a required prerequisite for securing Analytics.

The following procedure will install Apache and mod_ssl and set it to use a self-signed SSL certificate. You may also choose to purchase a SSL certificate signed by a third-party Certificate Authority or to generate your own SSL certificate.

Log in to the Analytics server as the root user, or as a user with superuser privileges. Install Apache and mod_ssl, configure Apache to start on server boot and start it for the first time:

yum -y install httpd
yum -y install mod_ssl
systemctl enable httpd
systemctl start httpd

You can check this was successful by opening the HTTP and HTTPS addresses in a web browser.

To support potential use of Internet Explorer 8 (IE8), Apache must be configured to strip out the "Pragma" statements from the headers of HTTP files. To do this, navigate to the following Apache configuration folder and edit the config file as follows:

cd /etc/httpd/conf
## Backup the existing config file
cp httpd.conf original_httpd.conf_original
## Edit the file
vi httpd.conf
## Add the following line right at the top of the file.
Header unset Pragma

Save the file and exit the editor.

Next, we configure SSL to add an internal proxy rule for Apache to proxy any request to the Analytics server and to turn on the Rewrite Engine. Navigate to the Apache SSL configuration folder and edit the SSL config file as follows:

cd /etc/httpd/conf.d
## Backup the existing config file
cp ssl.conf original_ssl.conf_original
## Edit the file
vi ssl.conf

Search for the string SSLCert, this should bring you to the "Server Certificate" section, which is immediately followed by other key and certificate options. Point these to the appropriate certificate and key. The CHAIN file is not required for a self signed certificate, or for a certificate signed directly by a private root CA, but should be the complete chain between the server certificate and the root certificate

SSLCertificateFile /etc/ssl/certs/SERVER.crt
SSLCertificateKeyFile /etc/ssl/private/SERVER.key

The CHAIN file in the same section is not required for a self signed certificate, nor for a certificate signed directly by a private root CA. In any other case it must contain the complete chain between the server certificate and the root certificate, the format is concatenated PEM.

SSLCertificateChainFile /etc/pki/tls/certs/CHAIN.crt

The last line of the file should be the closing tag </VirtualHost>. Add the following text just above this closing </VirtualHost> tag:

##Internal proxy rules instructing Apache to proxy any request to the
##Analytics server and data warehouse on 7070
ProxyPass /reports http://127.0.0.1:7070/reports
ProxyPassReverse /reports http://127.0.0.1:7070/reports
ProxyPass /etl http://127.0.0.1:7070/etl
ProxyPassReverse /etl http://127.0.0.1:7070/etl
##Turn on the RewriteEngine
RewriteEngine On
##Redirect any just / over to /reports
RewriteRule ^/+$ https://%{SERVER_NAME}:443/reports/ [R]

Save and close the ssl.conf file and then restart Apache.

systemctl restart httpd

Next we lockdown tomcat to localhost only so that the Analytics server will not respond to requests on its internal port (7070). An alternate solution is to simply close port 7070 altogether via firewall configuration. Note that if you are intending to use 3rd party tools with Jaspersoft you should NOT lockdown this port or make this server level config change.

Log in to the Analytics server as the root user, or as a user with superuser privileges and navigate to the server configuration file and edit it as follows:

cd /opt/zenoss_analytics/conf
## Make a backup of the server.xml file.
cp server.xml original_server.xml_original
## Edit the file
vi server.xml

Locate the following section in the file (/7070 in vi will locate it).

<Connector port="7070" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"/>

Change it to add in address="127.0.0.1" so that the section looks like the following:

<Connector port="7070" address="127.0.0.1" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"/>

Save and close the file and restart tomcat by restarting the service to pick up the changes.

service zenoss_analytics stop
service zenoss_analytics start

Installing the Analytics ZenETL ZenPack in Resource Manager

Analytics -related UI in Resource Manager, and ETL data extraction daemon services are provided by an Analytics -specific Resource Manager ZenPack, ZenETL. This is installed like any other ZenPack, and you should refer to the instructions specific to your version of Resource Manager for doing so if you are not already familiar with the process. In summary the process is the following:

Place ZenPacks.zenoss.ZenETL<version>.py2.7.egg in the /tmp/zenpacks directory on the Control Center master host, then perform the following:

chmod -R 777 /tmp/zenpacks
cd /tmp/zenpacks
serviced service run zope zenpack-manager install ZenPacks.zenoss.ZenETL<version>.py2.7.egg

To speed up performance data extractions, the following change should be made. The configuration option, tsd.query.skip_unresolved_tagvs should be set to True. This option can be set in the /opt/zenoss/etc/opentsdb/opentsdb.conf file that is editable in the "opentsdb reader" service in Control Center.

Finally, issue the following commands to stop and start Resource Manager so that zenperfetl, zenmodeletl, and zeneventetl initialize properly:

serviced service stop Zenoss.resmgr
## Wait for containers to stop
serviced service start Zenoss.resmgr

Updating the dim_date table

The default configuration of the dim_date table includes settings that cause performance ETL batches to fail. Use this procedure to update the table.

Download the fix archive file to your workstation, and then copy the file to the Analytics server host. This procedure assumes that you copy the file to the /tmp directory.
Log in to the Analytics server host as root or as a user with superuser privileges.
Change the permissions of the fix archive file to include read and execute for all users. For example, if the file is located in /tmp, enter the following command:
```
chmod o+rx /tmp/ANA-365-fix.tar.gz
```
Start a shell as user zenoss.
```
su - zenoss
```
Change directory to the Analytics bin directory.
```
cd /opt/zenoss_analytics/bin
```
Extract the contents of the fix archive file. For example, if the archive file is located in /tmp, enter the following command:
```
tar -xzvf /tmp/ANA-365-fix.tar.gz
```
Change directory to the newly-created subdirectory where the fix files are located.
```
cd ./analytics_fix
```
Start the fix script.
```
./fix_perf_batch_issues.sh
```
Exit the zenoss user shell.
```
exit
```
Restart the Analytics server.
```
systemctl restart zenoss_analytics
```

Connecting Analytics server to Resource Manager

To connect Resource Manager to the completed Analytics server installation is accomplished through the Resource Manager user interface. Log in to Resource Manager as an admin or "Manager" user.

Warning

For Resource Manager 5.x and 6.x only: Performance data extraction is performed by the Analytics server by calls to the Zenoss QueryService API. As such, a privileged user account dedicated to this purpose should be used. Before proceeding with the next paragraph, create such an account with a "Manager" role in Resource Manager and note the username and password. Suggested username is "analytics_etl". Note that the built in "admin" account cannot be used for this purpose.

Navigate to Reports > Configuration in the Resource Manager UI. The Analytics configuration page appears. Click the gear icon in the lower-left corner to display the Configure Analytics Base URLs page.

In the Internal URLs area, enter the URLs that Resource Manager services will use to communicate with Analytics. Note that if the two Resource Manager URLs are not displayed, you may need to shift-refresh your browser to drop the UI cache post-ZenETL installation. Do not proceed with configuration on this page unless both Resource Manager URLs are displayed correctly.

Paste the fully-qualified domain name (including SSL port number 443) for the Analytics server into both the "Internal Analytics" and "External Analytics" URL fields:

https://<analytics_server_fqdn>:443

Warning

For Resource Manager 5.x and 6.x only: In the QueryService area, enter the complete URL of the Resource Manager virtual host (that is, the URL for your Resource Manager UI) and the access credentials to the QueryService of this Resource Manager instance created above.

Click Submit to save the configuration. A success flare should confirm that configuration information was written to both Resource Manager and Analytics servers. If this process fails, make sure your DNS is valid in your client browser and on the Resource Manager master server (5.x and 6.x) or within all Zope containers (5.x and 6.x) and that environmental set up on the Analytics server was completed appropriately (that is, the firewall is disabled and appropriate ports are open).

Once this configuration has been successfully saved, the zenmodeletl, zeneventetl, and the zenperfetl service (all found in the Analytics ETL group) should be restarted in the Control Center UI so that the extractor services pick up the newly saved configuration for the Analytics server.

Warning

For Resource Manager 4.x only: At this point, you should update all hubs and collectors to push the new ZenPack code out. This can be achieved via the Advanced > Collectors screen in Resource Manager, or from the command line using dc-admin.

SSL Certificates are strictly verified

Warning

If you have enabled SSL on your Analytics server earlier, and are not using a publicly recognized certificate then you will need to install the root certificate of the CA into the Resource Manager image.

Have the Root certificate in PEM format in your local directory then:

Launch a Zope shell:

serviced service shell -i -s addRootCertificate zope bash

Navigate to the CA directory and create the certificate pem file:

cp /mnt/pwd/&lt;certificate&gt;.pem /etc/pki/ca-trust/source/anchors/

Update the certificate store (do not do by hand and please note there's no output):
```
update-ca-trust extract
```

Exit the container and commit the snapshot:

serviced snapshot commit fixAnalyticsCert

Delete the resulting snapshot tag (the output of the previous command):
```
serviced snapshot rm <tag>
```
Restart Resource Manager services:
```
serviced service restart zenoss.resmgr
```

Verifying installation success and correct ETL operation

Installation success should be immediately verified as follows. If any of the verification steps fail, refer to the Analytics configuration documentation for specific troubleshooting procedures:

Create a user account (or update an existing one) with BOTH the "Manager" and "ReportingUser" roles in Resource Manager and note the username and password. Log out of Resource Manager, and then log in to Resource Manager again using this user account. Navigate to Reporting > Advanced in the Resource Manager UI and successfully confirm that you were redirected to the Analytics UI and automatically logged into analytics. This confirms that single-sign on between Resource Manager and Analytics is working correctly. Note that the built-in Resource Manager "admin" account cannot be used for this purpose.
From the Reporting > Configuration screen in the Resource Manager UI, click the Add Batches button. Simply verify the MODEL, EVENTS and an entry matching the name of each collector (for Perf ETL extractors) are listed in the Extractor drop-down. This confirms that the extractor daemons have successfully registered with the Analytics server and that therefore the Analytics server will automatically schedule work for them on an ongoing basis.
Wait until at least midday local time on the day AFTER installation before proceeding with this step. This gives Analytics time to have scheduled and executed MODEL, EVENT, and PERFORMANCE extracts for each collector and to have aggregated that data which is necessary before any out-of-the-box data analytics capabilities in Analytics will show data. Then verify the following:
1. That all batches in Reporting > Configuration screen with an end time of within 15 mins prior to the current time are marked "COMPLETED".
2. Log in to Resource Manager using the test user account created in verification step 1 above. Navigate to Reporting > Advanced in the Resource Manager UI and then View > Repository in the Analytics UI. Verify that executing the following reports with a date range of "DAY-1" to "DAY-1" returns some data.
  - Zenoss > Current Reports > Event Reports > Events by Class Report
  - Zenoss > Current Reports > Device Performance > Availability, CPU Usage, and Memory Usage Exceptions Report. You may need to adjust all the input parameters specifically to match your environment to get results from this report, for example, Availability < 101%, Memory/CPU Usage > 0%.

Adding self monitoring and administration scripts

Download the latest version of the ZenPacks.zenoss.ZenossAnalyticsMonitoring ZenPack from delivery.zenoss.io. Once you have saved, unzip the .egg file and read the README.md. Follow all the instructions to get the ZenPack installed in Resource Manager and the Analytics server management scripts put in place in /opt/zenoss_analytics/bin on the Analytics server.

Performing initial performance data ETL configuration

Since monitoring templates are a function of the ZenPacks that provide them, the default set of aliases provided for any particular ZenPack is a function of what is provided by that ZenPack. Details of default aliases provided by a particular ZenPack are provided in the "Zenoss Analytics" section of the webpage for that ZenPack on ZenPack catalog.

Where it exists, such default supplied configuration should be thought of as definitive documentation of a fully comprehensive set of ETL configuration for all the data points monitored by that ZenPack and this ETL configuration will evolve with future ZenPack development. Thus, it is likely significant overkill to ETL all such data points to your Analytics data warehouse without any actual requirement for reporting on them. Doing so is likely to add more ETL load and use a lot of disk space on the Analytics server, which makes managing Analytics much harder than it would otherwise be.

The following procedure should be run immediately after initial install and repeated after every ZenPack upgrade or Resource Manager upgrade to re-audit the ETL configuration so that it is not inadvertently changed as a result of such activities.

If you are on Resource Manager 4.x, enter the following on the Resource Manager master:

su - zenoss

If you are on Resource Manager 5.x or 6.x, enter the following on the Control Center master:

serviced service attach zope/0
su - zenoss

Then, regardless of your Resource Manager version, change directory to the ZenETL ZenPack directory. In the following command, replace <version> with the current ZenETL ZenPack version:

cd /opt/zenoss/ZenPacks
cd ZenPacks.zenoss.ZenETL-<version>-py2.7.egg
cd ZenPacks/zenoss/ZenETL/bin

In this directory, there are two relevant python scripts:

dumpAliases.py
This script can be used to traverse your entire monitoring template hierarchy and dump out details of data points and aliases.
Run dumpAliases.py --help to read the detailed help.
The script takes one required parameter --aliases= with a value of either with or without. The former dumps details of all aliases on data points, the later dumps details of all data points with no aliases.
The optional parameter -f or --outputFile can use to specify a filename to write the information to. If you exclude this file, the default filename of aliases.txt is used. The output format of this file is in the exact required input format for manageAliases.py.
manageAliases.py
This script can be used to bulk or remove aliases from data points in your monitoring template hierarchy. Run manageAliases.py --help to read the detailed help on options. Run manageAliases.py --printhelp to read the detailed help on input file format and output results.
The script takes one required parameter --action= with a value of either add or remove. The former adds aliases to data points, the later removes existing aliases from data points.
By default it is "read-only", that is it only reports what it could/would do regarding changing aliases on monitoring templates. You must call it with the --commit option to actually make changes effective on your monitoring templates.

Use these scripts as follows to get your ETL configuration to an initial known state with metrics for device-level CPU and memory usage, and component-level filesystem and IP interface metrics ETL to Analytics. This will allow out-of-the-box sample reports to execute successfully. In the following steps, replace <YYYYMMDD> with today's date.

Dump out the current state of aliases in your install:

./dumpAliases.py --aliases=with --outputFile=original_aliases_<YYYYMMDD>.txt

Make a copy of this file and keep it somewhere safe for future reference:
```
scp original_aliases_<YYYYMMDD>.txt myuser@myserver:/pathtosafeplace/
```
Use this file to remove ALL aliases currently in place:
```
./manageAliases.py --action=remove --inputFile=original_aliases_<YYYYMMDD>.txt 2>&1 | tee removal_test.log
```
Check that this would remove all aliases successfully. The following grep should return no alias lines:
```
grep -iv "^-" removal_test.log | grep -v ControlCenter
```
If not, run manageAliases.py --printhelp to get information of what the error codes at the start of each line indicate.

If successful, repeat with --commit to actually make the changes:
```
./manageAliases.py --action=remove --inputFile=original_aliases_<YYYYMMDD>.txt --commit 2>&1 | tee aliases_removed_<YYYYMMDD>.log
```
Confirm this was successful, that is you now have no aliases at all:
```
./dumpAliases.py --aliases=with
```
A file named aliases.txt with no contents should have been created. To check that no contents were created, execute the following grep command:
```
grep -v ControlCenter aliases.txt
```

Create a desired subset of aliases from this file for the above mentioned device- and component-level metrics:

head -1 original_aliases_<YYYYMMDD>.txt > current_aliases_<YYYYMMDD>.txt \
  | grep -E "\|(cpu__pct|mem__pct|fs__pct|in__pct|out__pct)\|" original_aliases_<YYYYMMDD>.txt \
  >> current_aliases_<YYYYMMDD>.txt

Keep a copy of this file safe for future reference. It documents what your ETL configuration is always supposed to be. Zenoss highly recommends you put it under configuration management, and incrementally add to it over time as you add more data to the data warehouse to meet Analytics reporting needs.
Add these aliases to your monitoring templates:
```
./manageAliases.py --action=add --inputFile=current_aliases_<YYYYMMDD>.txt  2>&1 | tee add_test.log
```
Check that this would add all aliases successfully. The following grep should return no alias lines:
```
grep -iv "^+" add_test.log | grep -v ControlCenter
```
If not, run manageAliases.py --printhelp to get information of what the error codes at the start of each line indicate.

If successful, repeat with --commit to actually make the changes:
```
./manageAliases.py --action=add --inputFile=current_aliases_<YYYYMMDD>.txt  --commit 2>&1 | tee aliases_added_<YYYYMMDD>.log
```
Confirm this was successful, that is you now only have exactly the aliases you added:
```
./dumpAliases.py --aliases=with --outputFile=aliases.txt
sort -u aliases.txt > a
sort -u current_aliases_<YYYYMMDD>.txt > b
diff a b
```
The diff should show no differences between the two files.

Verifying your ETL configuration

To verify that your ETL configuration is correct, perform the following:

Log in to the Analytics database directly and execute the following query:

select distinct metric_name from reporting.meta_metric order by metric_name;

The aliases you created will appear in the returned list after the next scheduled zenperfetl batch post aliases add has successfully completed for a collector that has devices monitored using the monitoring template that aliases have been added to.