Skip to content

Configure the ZooKeeper ensemble

Use the following procedures to configure a ZooKeeper ensemble (cluster) for a multi-host Control Center deployment that includes a minimum of three hosts.

ZooKeeper and Control Center

Control Center relies on Apache ZooKeeper to distribute and manage application services. ZooKeeper maintains the definitions of each service and the list of services assigned to each host. The scheduler, which runs on the master host, determines assignments and sends them to the ZooKeeper node that is serving as the ensemble leader. The leader replicates the assignments to the other ensemble nodes, so that the other nodes can assume the role of leader if the leader node fails.

All Control Center hosts retrieve assignments and service definitions from the ZooKeeper ensemble leader and then start services in Docker containers as required. So, the Control Center configuration files of all Control Center hosts must include a definition for the SERVICED_ZK variable, which specifies the ZooKeeper endpoints of the ensemble nodes. Additional variables are required on ensemble nodes.

A ZooKeeper ensemble requires a minimum of three nodes, which is sufficient for most environments. An odd number of nodes is recommended and an even number of nodes is strongly discouraged. A five-node ensemble improves failover protection during maintenance windows but larger ensembles yield no benefits.

The Control Center master host is always an ensemble node. All ensemble nodes should be on the same subnet.

ZooKeeper security

The default configuration of Zookeeper is insecure, in two ways.

Default Zookeeper configuration Optional Control Center configuration
During the leader-election phase, communication among ensemble servers is not authenticated.

Set the SERVICED_ISVCS_ZOOKEEPER_USERNAME and SERVICED_ISVCS_ZOOKEEPER_PASSWD variables.

These secure communication during leader election.
Any user can access Zookeeper data nodes.
  1. Set the SERVICED_ZOOKEEPER_ACL_USER and SERVICED_ZOOKEEPER_ACL_PASSWD variables.
    These secure all but the /zookeeper and /zookeeper/quota data nodes.
  2. Optionally, secure the /zookeeper and /zookeeper/quota data nodes.

Typically, these variables are set during the installation process. You can also add them to an existing deployment.

Understanding the configuration process

The procedures in this section instruct you to create temporary variables that are used as building blocks, to construct Control Center configuration variables accurately. You append the Control Center variables to /etc/default/serviced, and then edit the file to move the variables to more appropriate locations.

The most important temporary variables specify the IP address or hostname of each host in the ZooKeeper ensemble. The following table identifies these important variables, the names and values of which must be identical on every Control Center host.

Variable name Placeholder value Actual value
node1 Master The IP address or hostname of the master host.
node2 Delegate-A The IP address or hostname of delegate host A.
node3 Delegate-B The IP address or hostname of delegate host B.

All ensemble hosts should be on the same subnet.

ZooKeeper variables

The variables in the following table are set only on ZooKeeper ensemble nodes, except SERVICED_ZK, which must be identical on all Control Center hosts.

Variable Where to set
SERVICED_ISVCS_START ZooKeeper ensemble nodes
SERVICED_ISVCS_ZOOKEEPER_ID ZooKeeper ensemble nodes
SERVICED_ISVCS_ZOOKEEPER_QUORUM ZooKeeper ensemble nodes
SERVICED_ZK All Control Center hosts
SERVICED_ZK_SESSION_TIMEOUT ZooKeeper ensemble nodes

Example multi-host ZooKeeper configuration

This example shows the ZooKeeper variables in the /etc/default/serviced configuration file of each host in a 4-node Control Center deployment. For convenience, the relevant settings for each node or host are also included in subsequent procedures.

Master host and ZooKeeper ensemble node, 198.51.100.135:

SERVICED_ISVCS_ZOOKEEPER_ID=1
SERVICED_ZK=198.51.100.135:2181,198.51.100.136:2181,198.51.100.137:2181
SERVICED_ISVCS_ZOOKEEPER_QUORUM=1@0.0.0.0:2888:3888,2@198.51.100.136:2888:3888,3@198.51.100.137:2888:3888
SERVICED_ZK_SESSION_TIMEOUT=15

Delegate host and ZooKeeper ensemble node, 198.51.100.136:

SERVICED_ISVCS_START=zookeeper
SERVICED_ISVCS_ZOOKEEPER_ID=2
SERVICED_ZK=198.51.100.135:2181,198.51.100.136:2181,198.51.100.137:2181
SERVICED_ISVCS_ZOOKEEPER_QUORUM=1@198.51.100.135:2888:3888,2@0.0.0.0:2888:3888,3@198.51.100.137:2888:3888
SERVICED_ZK_SESSION_TIMEOUT=15

Delegate host and ZooKeeper ensemble node, 198.51.100.137:

SERVICED_ISVCS_START=zookeeper
SERVICED_ISVCS_ZOOKEEPER_ID=3
SERVICED_ZK=198.51.100.135:2181,198.51.100.136:2181,198.51.100.137:2181
SERVICED_ISVCS_ZOOKEEPER_QUORUM=1@198.51.100.135:2888:3888,2@198.51.100.136:2888:3888,3@0.0.0.0:2888:3888
SERVICED_ZK_SESSION_TIMEOUT=15

Delegate host, 198.51.100.138:

SERVICED_ZK=198.51.100.135:2181,198.51.100.136:2181,198.51.100.137:2181

Configuring the master host as a ZooKeeper node

This procedure configures the Control Center master host as a node in a ZooKeeper ensemble.

  1. Log in to the master host as root or as a user with superuser privileges.

  2. Define the IP address variables for each node in the ZooKeeper ensemble.Replace Master with the IP address or hostname of the Control Center master host, and replace Delegate-A and Delegate-B with the IP addresses or hostnames of the delegate hosts to include in the ensemble:

    node1=Master
node2=Delegate-A
node3=Delegate-B

  3. Set the ZooKeeper node ID to 1.

    echo "SERVICED_ISVCS_ZOOKEEPER_ID=1" >> /etc/default/serviced

  4. Specify the nodes in the ZooKeeper ensemble.You can copy the following text and paste it in your console:

    echo "SERVICED_ZK=${node1}:2181,${node2}:2181,${node3}:2181" >> /etc/default/serviced

  5. Specify the nodes in the ZooKeeper quorum. ZooKeeper requires a unique quorum definition for each node in its ensemble. To achieve this, replace the IP address or hostname of the master host with 0.0.0.0.You can copy the following text and paste it in your console:

    q1="1@0.0.0.0:2888:3888"
q2="2@${node2}:2888:3888"
q3="3@${node3}:2888:3888"
echo "SERVICED_ISVCS_ZOOKEEPER_QUORUM=${q1},${q2},${q3}" >> /etc/default/serviced

  6. Specify the timeout for inactive connections.You can copy the following text and paste it in your console:

    echo "SERVICED_ZK_SESSION_TIMEOUT=15" >> /etc/default/serviced

  7. Clean up the Control Center configuration file.

    1. Open /etc/default/serviced in a text editor.
    2. Navigate to the end of the file, and cut the line that contains the SERVICED_ZK variable declaration at that location.
    3. Locate the original SERVICED_ZK variable declaration, and then paste the cut line immediately below it.
    4. Navigate to the end of the file, and cut the line that contains the SERVICED_ISVCS_ZOOKEEPER_ID variable declaration at that location.
    5. Locate the original SERVICED_ISVCS_ZOOKEEPER_ID variable declaration, and then paste the cut line immediately below it.
    6. Navigate to the end of the file, and cut the line that contains the SERVICED_ISVCS_ZOOKEEPER_QUORUM variable declaration at that location.
    7. Locate the original SERVICED_ISVCS_ZOOKEEPER_QUORUM variable declaration, and then paste the cut line immediately below it.
    8. Navigate to the end of the file, and cut the line that contains the SERVICED_ZK_SESSION_TIMEOUT variable declaration at that location.
    9. Locate the original SERVICED_ZK_SESSION_TIMEOUT variable declaration, and then paste the cut line immediately below it.
    10. Save the file, and then close the editor.

  8. Verify the ZooKeeper environment variables.

    grep -E '^\b*SERVICED' /etc/default/serviced | grep -E '_Z(OO|K)'

    The following example shows the environment variables for a master host with IP address 198.51.100.135.

    SERVICED_ZK=198.51.100.135:2181,198.51.100.136:2181,198.51.100.137:2181
SERVICED_ISVCS_ZOOKEEPER_ID=1
SERVICED_ISVCS_ZOOKEEPER_QUORUM=1@0.0.0.0:2888:3888,2@198.51.100.136:2888:3888,3@198.51.100.137:2888:3888
SERVICED_ZK_SESSION_TIMEOUT=15

Configuring delegate host A as a ZooKeeper node

Use this procedure to configure the delegate host designated as Delegate-A as a ZooKeeper node.

  1. Log in to the delegate host as root or as a user with superuser privileges.

  2. Define the IP address variables for each node in the ZooKeeper ensemble.Replace Master with the IP address or hostname of the Control Center master host, and replace Delegate-A and Delegate-B with the IP addresses or hostnames of the delegate hosts to include in the ensemble:

    node1=Master
node2=Delegate-A
node3=Delegate-B

  3. Set the ID of this node in the ZooKeeper ensemble.

    echo "SERVICED_ISVCS_ZOOKEEPER_ID=2" >> /etc/default/serviced

  4. Specify the nodes in the ZooKeeper ensemble.You can copy the following text and paste it in your console:

    echo "SERVICED_ZK=${node1}:2181,${node2}:2181,${node3}:2181" >> /etc/default/serviced

  5. Specify the nodes in the ZooKeeper quorum. ZooKeeper requires a unique quorum definition for each node in its ensemble. To achieve this, replace the IP address or hostname of delegate host A with 0.0.0.0.You can copy the following text and paste it in your console:

    q1="1@${node1}:2888:3888"
q2="2@0.0.0.0:2888:3888"
q3="3@${node3}:2888:3888"
echo "SERVICED_ISVCS_ZOOKEEPER_QUORUM=${q1},${q2},${q3}" >> /etc/default/serviced

  6. Specify the timeout for inactive connections.You can copy the following text and paste it in your console:

    echo "SERVICED_ZK_SESSION_TIMEOUT=15" >> /etc/default/serviced

  7. Configure Control Center to start the ZooKeeper service.You can copy the following text and paste it in your console:

    echo "SERVICED_ISVCS_START=zookeeper" >> /etc/default/serviced

  8. Clean up the Control Center configuration file.

    1. Open /etc/default/serviced in a text editor.
    2. Navigate to the end of the file, and cut the line that contains the SERVICED_ZK variable declaration at that location.
    3. Locate the original SERVICED_ZK variable declaration, and then paste the cut line immediately below it.
    4. Comment the original SERVICED_ZK declaration, which references only the master host. Insert the number sign character (#) immediately in front of the original SERVICED_ZK variable.
    5. Navigate to the end of the file, and cut the line that contains the SERVICED_ISVCS_ZOOKEEPER_ID variable declaration at that location.
    6. Locate the original SERVICED_ISVCS_ZOOKEEPER_ID variable declaration, and then paste the cut line immediately below it.
    7. Navigate to the end of the file, and cut the line that contains the SERVICED_ISVCS_ZOOKEEPER_QUORUM variable declaration at that location.
    8. Locate the original SERVICED_ISVCS_ZOOKEEPER_QUORUM variable declaration, and then paste the cut line immediately below it.
    9. Navigate to the end of the file, and cut the line that contains the SERVICED_ZK_SESSION_TIMEOUT variable declaration at that location.
    10. Locate the original SERVICED_ZK_SESSION_TIMEOUT variable declaration, and then paste the cut line immediately below it.
    11. Navigate to the end of the file, and cut the line that contains the SERVICED_ISVCS_START variable declaration at that location.
    12. Locate the original SERVICED_ISVCS_START variable declaration, and then paste the cut line immediately below it.
    13. Save the file, and then close the editor.

  9. Verify the ZooKeeper environment variables.

    grep -E '^[[:space:]]*SERVICED' /etc/default/serviced | grep -E '(CS_ZO|_ZK|CS_ST)'

    The following example shows the environment variables for a delegate host with IP address 198.51.100.136.

    SERVICED_ZK=198.51.100.135:2181,198.51.100.136:2181,198.51.100.137:2181
SERVICED_ISVCS_START=zookeeper
SERVICED_ISVCS_ZOOKEEPER_ID=2
SERVICED_ISVCS_ZOOKEEPER_QUORUM=1@198.51.100.135:2888:3888,2@0.0.0.0:2888:3888,3@198.51.100.137:2888:3888
SERVICED_ZK_SESSION_TIMEOUT=15

Configuring delegate host B as a ZooKeeper node

Use this procedure to configure the delegate host designated as Delegate-B as a ZooKeeper node.

  1. Log in to the delegate host as root or as a user with superuser privileges.

  2. Define the IP address variables for each node in the ZooKeeper ensemble.Replace Master with the IP address or hostname of the Control Center master host, and replace Delegate-A and Delegate-B with the IP addresses or hostnames of the delegate hosts to include in the ensemble:

    node1=Master
node2=Delegate-A
node3=Delegate-B

  3. Set the ID of this node in the ZooKeeper ensemble.

    echo "SERVICED_ISVCS_ZOOKEEPER_ID=3" >> /etc/default/serviced

  4. Specify the nodes in the ZooKeeper ensemble.You can copy the following text and paste it in your console:

    echo "SERVICED_ZK=${node1}:2181,${node2}:2181,${node3}:2181" >> /etc/default/serviced

  5. Specify the nodes in the ZooKeeper quorum. ZooKeeper requires a unique quorum definition for each node in its ensemble. To achieve this, replace the IP address or hostname of delegate host B with 0.0.0.0.You can copy the following text and paste it in your console:

    q1="1@${node1}:2888:3888"
q2="2@${node2}:2888:3888"
q3="3@0.0.0.0:2888:3888"
echo "SERVICED_ISVCS_ZOOKEEPER_QUORUM=${q1},${q2},${q3}" >> /etc/default/serviced

  6. Specify the timeout for inactive connections.You can copy the following text and paste it in your console:

    echo "SERVICED_ZK_SESSION_TIMEOUT=15" >> /etc/default/serviced

  7. Configure Control Center to start the ZooKeeper service.You can copy the following text and paste it in your console:

    echo "SERVICED_ISVCS_START=zookeeper" >> /etc/default/serviced

  8. Clean up the Control Center configuration file.

    1. Open /etc/default/serviced in a text editor.
    2. Navigate to the end of the file, and cut the line that contains the SERVICED_ZK variable declaration at that location.
    3. Locate the original SERVICED_ZK variable declaration, and then paste the cut line immediately below it.
    4. Comment the original SERVICED_ZK declaration, which references only the master host. Insert the number sign character (#) immediately in front of the original SERVICED_ZK variable.
    5. Navigate to the end of the file, and cut the line that contains the SERVICED_ISVCS_ZOOKEEPER_ID variable declaration at that location.
    6. Locate the original SERVICED_ISVCS_ZOOKEEPER_ID variable declaration, and then paste the cut line immediately below it.
    7. Navigate to the end of the file, and cut the line that contains the SERVICED_ISVCS_ZOOKEEPER_QUORUM variable declaration at that location.
    8. Locate the original SERVICED_ISVCS_ZOOKEEPER_QUORUM variable declaration, and then paste the cut line immediately below it.
    9. Navigate to the end of the file, and cut the line that contains the SERVICED_ZK_SESSION_TIMEOUT variable declaration at that location.
    10. Locate the original SERVICED_ZK_SESSION_TIMEOUT variable declaration, and then paste the cut line immediately below it.
    11. Navigate to the end of the file, and cut the line that contains the SERVICED_ISVCS_START variable declaration at that location.
    12. Locate the original SERVICED_ISVCS_START variable declaration, and then paste the cut line immediately below it.
    13. Save the file, and then close the editor.

  9. Verify the ZooKeeper environment variables.

    grep -E '^[[:space:]]*SERVICED' /etc/default/serviced | grep -E '(CS_ZO|_ZK|CS_ST)'

    The following example shows the environment variables for a delegate host with IP address 198.51.100.137.

    SERVICED_ZK=198.51.100.135:2181,198.51.100.136:2181,198.51.100.137:2181
SERVICED_ISVCS_START=zookeeper
SERVICED_ISVCS_ZOOKEEPER_ID=3
SERVICED_ISVCS_ZOOKEEPER_QUORUM=1@198.51.100.135:2888:3888,2@198.51.100.136:2888:3888,3@0.0.0.0:2888:3888
SERVICED_ZK_SESSION_TIMEOUT=15

Importing the Docker image for ZooKeeper

Perform the steps in Downloading and staging required files before performing this procedure.

Use this procedure to import the Docker image for ZooKeeper on delegate hosts A and B. This procedure is not necessary on the master host.

  1. Log in to the host as root or as a user with superuser privileges.

  2. Change directory to /root.

    cd /root

  3. Extract the ZooKeeper image.

    yes | ./install-zenoss-isvcs-zookeeper_v*.run

  4. Optional: Delete the archive file, if desired.

    rm -i ./install-zenoss-isvcs-zookeeper_v*.run

Starting a ZooKeeper ensemble

Use this procedure to start a ZooKeeper ensemble. This procedures uses the nc utility to query ensemble hosts. If nc is not available, you can use telnet with interactive ZooKeeper commands.

The window of time for starting a ZooKeeper ensemble is relatively short. The goal of this procedure is to restart Control Center on each ensemble node at about the same time, so that each node can participate in electing the leader.

  1. Log in to the Control Center master host as root or as a user with superuser privileges.

  2. In a separate window, log in to the second node of the ZooKeeper ensemble (Delegate-A) as root or as a user with superuser privileges.

  3. In a different window, log in to the third node of the ZooKeeper ensemble (Delegate-B) as root or as a user with superuser privileges.

  4. On all ensemble hosts, stop and start serviced.

    systemctl stop serviced && systemctl start serviced

  5. On the master host, check the status of the ZooKeeper ensemble.

    1. Attach to the container of the ZooKeeper service.

      docker exec -it serviced-isvcs_zookeeper /bin/bash

    2. Define IP address variables for each node in the ZooKeeper ensemble.Replace Master with the IP address or hostname of the Control Center master host, and replace Delegate-A and Delegate-B with the IP addresses or hostnames of the delegate hosts in the ensemble:

      node1=Master
node2=Delegate-A
node3=Delegate-B

    3. Query the master host and identify its role in the ensemble.

      { echo stats; sleep 1; } | nc $node1 2181 | grep Mode

      The result includes leader or follower.

    4. Query delegate host A and identify its role in the ensemble.

      { echo stats; sleep 1; } | nc $node2 2181 | grep Mode

    5. Query delegate host B and identify its role in the ensemble.

      { echo stats; sleep 1; } | nc $node3 2181 | grep Mode

    6. Detach from the container of the ZooKeeper service.

      exit

    If none of the hosts reports that it is the ensemble leader within a few minutes of starting serviced, reboot the hosts.

    Once the ZooKeeper quorum is running, update and start the remaining delegate hosts.

Updating and starting other hosts

The default configuration of delegate hosts sets the value of the SERVICED_ZK variable to the master host only. Use this procedure to update the setting to include all of the hosts in the ZooKeeper ensemble. Perform this procedure on each host in your Control Center deployment that is not a ZooKeeper ensemble node.

  1. Log in to the delegate host as root or as a user with superuser privileges.

  2. Define the IP address variables for each node in the ZooKeeper ensemble.Replace Master with the IP address or hostname of the Control Center master host, and replace Delegate-A and Delegate-B with the IP addresses or hostnames of the delegate hosts to include in the ensemble:

    node1=Master
node2=Delegate-A
node3=Delegate-B

  3. Specify the nodes in the ZooKeeper ensemble.You can copy the following text and paste it in your console:

    echo "SERVICED_ZK=${node1}:2181,${node2}:2181,${node3}:2181" >> /etc/default/serviced

  4. Update the variable.

    1. Open /etc/default/serviced in a text editor.
    2. Navigate to the end of the file, and cut the line that contains the SERVICED_ZK variable declaration at that location. The value of this declaration specifies three endpoints.
    3. Locate the SERVICED_ZK variable near the beginning of the file, and then delete the line it is on. The value is just the master host endpoint.
    4. Paste the SERVICED_ZK variable declaration from the end of the file in the location of the just-deleted declaration.
    5. Save the file, and then close the editor.

  5. Verify the setting.

    grep -E '^[[:space:]]*SERVICED_ZK' /etc/default/serviced

    The following example shows the environment variable for a delegate host that is not a node in the ZooKeeper ensemble:

    SERVICED_ZK=198.51.100.135:2181,198.51.100.136:2181,198.51.100.137:2181

  6. Restart Control Center.

    systemctl restart serviced

Optional: Secure remaining data nodes

When you set the SERVICED_ZOOKEEPER_ACL_USER and SERVICED_ZOOKEEPER_ACL_PASSWD variables, Control Center is able to secure all but the /zookeeper and /zookeeper/quota data nodes; use this procedure to secure them. For more information, see Zookeeper security.

This procedure secures the /zookeeper and /zookeeper/quota data nodes as long as the /opt/serviced/var/isvcs/zookeeper directory is intact (not deleted and recreated).

Perform this procedure on each Control Center host in your deployment, not just ensemble hosts.

Follow these steps:

  1. Log in to the Control Center host as root or as a user with superuser privileges.

  2. Display the Zookeeper ACL security variables.

    grep -E '^[[:space:]]*.*KEEPER_ACL.*=' /etc/default/serviced

    If the command returns no result, stop this procedure and perform the procedure for updating an existing deployment instead.

  3. Verify that Control Center is running.

    systemctl status serviced || systemctl start serviced

  4. Attach to the Zookeeper container.

    docker exec -it serviced-isvcs_zookeeper bash

  5. Start the Zookeeper CLI.

    sh /opt/apache-zookeeper-3.5.5-bin/bin/zkCli.sh

  6. Authorize the credentials you used for the Zookeeper ACL variables (step 2).

    addauth digest <USERNAME>:<PASSWORD>

  7. Display the hashed password.

    getAcl /docker

    If /docker does not return a result, repeat the command with /net, /pools, /scheduler, or /storage. The result should look similar to the following example:

    'digest,'<USERNAME>:u3v8E1ESKqkXNn827qd3HrGCBjI=
: cdrwa

  8. Stop the Zookeeper CLI.

    quit

  9. Start the Zookeeper CLI again.

    sh /opt/apache-zookeeper-3.5.5-bin/bin/zkCli.sh

  10. Set the credentials for the /zookeeper data node.

    setAcl /zookeeper digest:<USERNAME>:<HASH>:crdwa

  11. Set the credentials for the /zookeeper/quota data node.

    setAcl /zookeeper/quota digest:<USERNAME>:<HASH>:crdwa

  12. Stop the Zookeeper CLI.

    quit

  13. Exit the Zookeeper container.

    exit