Emergency shutdown of services

Control Center monitors each service's short-term storage usage trends and current usage levels. When Control Center predicts that a service is about to exhaust storage space, it initiates an automatic emergency shutdown of the service. By shutting down while enough space is available to perform recovery operations, Control Center minimizes the risk of data corruption.

Emergency shutdown is performed for services that are in a resource pool that has DFS access. Services that are in pools that do not have DFS permissions and do not write to the DFS continue running.

Control Center displays thin pool and DFS information in the following graphs in the browser interface:

• On the Applications tab, the Thin Pool Usage graph shows used and available bytes for the thin pool.
• On the Applications page for each service, the DFS Usage graph shows used and available bytes for the DFS.

By comparing usage to the available storage and the amount of space that must be reserved, Control Center determines when a service must be shut down before filling the thin pool or DFS, and initiates the emergency shutdown. The browser interface identifies services in the emergency shutdown state, as does issuing the serviced service status command.

To minimize data loss, Control Center shuts down services in the following order: databases; services that cannot be recovered; indices and services that are difficult to recover; any other services. Services in emergency shutdown status cannot be restarted until the underlying cause of the shutdown is resolved.

To resolve an emergency shutdown:

1. Examine the service that was shut down to determine why it was using excessive storage and correct the issue. For example:
   • If an application was writing a large amount of performance data to the tenant device, add space to the device. See Control Center application data storage requirements.
   • If too many snapshots are stored on the device, delete those that you no longer need. See Creating snapshots and rolling back.
   • If a usage anomaly might have occurred, wait for usage levels to return to normal.
2. Clear the emergency shutdown flags.
3. Start the service by using the browser interface or command line interface. Control Center starts services in the reverse order of shutdown.

Resetting emergency shutdown flags

After resolving the issue that caused an emergency shutdown, use this procedure to determine service status, and then restart the service.

1. Log in to the Control Center master host as root or as a user with superuser privileges.

2. Check services for the emergency shutdown flag:

   for s in $(serviced service list --show-fields ServiceID \
     | grep -v ServiceID); do serviced service list $s \
     | grep EmergencyShutdown\"; done
   

   "True" indicates an emergency shutdown.

3. Clear the emergency-shutdown flag for a service or the entire application:

   serviced service clear-emergency SERVICEID
   

   serviced service clear-emergency APPLICATION_ID
   

4. Check services for the emergency shutdown flag:

   for s in $(serviced service list --show-fields ServiceID \
     | grep -v ServiceID); do serviced service list $s \
     | grep EmergencyShutdown\"; done
   

   "False" indicates that the flags have been cleared.

5. Start the service:

   serviced service start SERVICEID