Skip to content

Troubleshooting

Analytics writes messages about failures to the following log files:

  • /opt/zenoss_analytics/logs/etl.log (ETL batch messages)
  • /opt/zenoss_analytics/logs/catalina.log (Tomcat log, also available through journalctl)

Too few CPU cores

Error messages similar to the following can appear in the ETL log file when the Analytics server host has fewer than 8 CPU cores.

2023-06-02T08:38:10,302 INFO  c.z.reporting.WebApplicationExceptionMapper [http-nio-7070-exec-10] javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error: com.zenoss.reporting.batch.BatchResource.update() (HTTP status code: 500)
2023-06-02T08:38:39,585 INFO  c.z.reporting.batch.BatchResource [http-nio-7070-exec-6] Batch updated  batch=164 tenant="02cd2b16-5f6e-11ed-8da9-0242ac11001a" extractor="PERFORMANCE" state=CONFIGURING
2023-06-02T08:38:40,261 WARN  o.m.jdbc.message.server.ErrorPacket [http-nio-7070-exec-4] Error: 1205-HY000: Lock wait timeout exceeded; try restarting transaction
2023-06-02T08:38:40,261 ERROR o.h.engine.jdbc.spi.SqlExceptionHelper [http-nio-7070-exec-4] (conn=1671) Lock wait timeout exceeded; try restarting transaction
2023-06-02T08:38:40,261 ERROR c.z.reporting.batch.BatchResource [http-nio-7070-exec-4] Failed to update batch  batch=149 tenant="02cd2b16-5f6e-11ed-8da9-0242ac11001a" extractor="PERFORMANCE"
javax.persistence.PessimisticLockException: could not execute statement
        at org.hibernate.internal.ExceptionConverterImpl.wrapLockException(ExceptionConverterImpl.java:273) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]

……

2023-06-02T08:56:09,612 WARN  o.m.jdbc.message.server.ErrorPacket [quartzScheduler_Worker-5] Error: 1205-HY000: Lock wait timeout exceeded; try restarting transaction
2023-06-02T08:56:09,612 ERROR c.z.reporting.schedule.CancelBatchesJob [quartzScheduler_Worker-5] call to rotate_fct_partitions failed: java.sql.SQLException: (conn=1688) Lock wait timeout exceeded; try restarting transaction
2023-06-02T08:58:03,575 INFO  o.s.scheduling.quartz.LocalDataSourceJobStore [QuartzScheduler_quartzScheduler-NON_CLUSTERED_MisfireHandler] Handling 1 trigger(s) that missed their scheduled fire-time.

2023-06-02T10:13:27,627 INFO  o.s.scheduling.quartz.LocalDataSourceJobStore [QuartzScheduler_quartzScheduler-NON_CLUSTERED_MisfireHandler] Handling 1 trigger(s) that missed their scheduled fire-time.
2023-06-02T10:14:17,628 WARN  o.m.jdbc.message.server.ErrorPacket [quartzScheduler_Worker-8] Error: 1205-HY000: Lock wait timeout exceeded; try restarting transaction
2023-06-02T10:15:20,893 INFO  o.s.scheduling.quartz.LocalDataSourceJobStore [QuartzScheduler_quartzScheduler-NON_CLUSTERED_MisfireHandler] Handling 2 trigger(s) that missed their scheduled fire-time.

Too little main memory

Error messages similar to the following can appear in the ETL log file when the Analytics server host has less than 32GB of main memory.

2023-05-11T06:03:34,739 INFO  c.z.reporting.schedule.LoadModelJob [quartzScheduler_Worker-8] Updated batch state  extractor=2 type=MODEL batch=830 state=LOADING
2023-05-11T06:04:54,776 WARN  c.z.hikari.pool.ProxyConnection [quartzScheduler_Worker-8] etlHikariCP - Connection org.mariadb.jdbc.Connection@1be71a62 marked as broken because of SQLSTATE(08000), ErrorCode(-1)
java.sql.SQLNonTransientConnectionException: (conn=207) Socket error
        at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:293) ~[mariadb-java-client-3.1.2.jar:?]
     …….
              at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75) [spring-context-support-5.3.27.jar:5.3.27]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.3.2.jar:?]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.3.2.jar:?]
Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
        at org.mariadb.jdbc.client.socket.impl.PacketReader.readReusablePacket(PacketReader.java:68) ~[mariadb-java-client-3.1.2.jar:?]
        at org.mariadb.jdbc.message.ClientMessage.readPacket(ClientMessage.java:115) ~[mariadb-java-client-3.1.2.jar:?]
        at org.mariadb.jdbc.message.client.PrepareExecutePacket.readPacket(PrepareExecutePacket.java:184) ~[mariadb-java-client-3.1.2.jar:?]
        at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:855) ~[mariadb-java-client-3.1.2.jar:?]
        ... 22 more
2023-05-11T06:04:55,281 ERROR o.h.engine.jdbc.spi.SqlExceptionHelper [quartzScheduler_Worker-8] (conn=207) Socket error
2023-05-11T06:04:55,435 ERROR o.h.engine.jdbc.spi.SqlExceptionHelper [quartzScheduler_Worker-8] Connection is closed
2023-05-11T06:04:55,440 WARN  c.z.reporting.MetaSetting [quartzScheduler_Worker-8] Using default value. Unable to parse meta_setting value  name='keep_batch_file_on_error' value='null' default='false' type=class java.lang.Boolean error=org.hibernate.exception.GenericJDBCException: could not prepare statement
2023-05-11T06:04:55,462 ERROR o.h.engine.jdbc.spi.SqlExceptionHelper [quartzScheduler_Worker-8] Connection is closed
2023-05-11T06:04:55,462 WARN  c.z.reporting.MetaSetting [quartzScheduler_Worker-8] Using default value. Unable to parse meta_setting value  name='save_all_etl_logs' value='null' default='false' type=class java.lang.Boolean error=org.hibernate.exception.GenericJDBCException: could not prepare statement
2023-05-11T06:04:55,546 ERROR c.z.reporting.schedule.LoadModelJob [quartzScheduler_Worker-8] Error loading data  extractor=2 type=MODEL batch=830 file="/opt/zenoss_analytics/temp/MODEL_3282247214843475602.csv.gz" error="Unable to rollback against JDBC Connection" cause{}="Connection is closed"
2023-05-11T06:04:55,572 ERROR c.z.reporting.schedule.LoadModelJob [quartzScheduler_Worker-8] Failed to process batch  extractor=2 type=MODEL batch=830
org.hibernate.TransactionException: JDBC begin transaction failed:
        at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:78) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
 ……….
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.3.2.jar:?]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.3.2.jar:?]
Caused by: java.sql.SQLException: Connection is closed
        at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection$1.invoke(ProxyConnection.java:467) ~[HikariCP-java7-2.4.13.jar:?]
        at com.sun.proxy.$Proxy74.setAutoCommit(Unknown Source) ~[?:?]
        at com.zaxxer.hikari.pool.ProxyConnection.setAutoCommit(ProxyConnection.java:377) ~[HikariCP-java7-2.4.13.jar:?]
        at com.zaxxer.hikari.pool.HikariProxyConnection.setAutoCommit(HikariProxyConnection.java) ~[HikariCP-java7-2.4.13.jar:?]
        at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:72) ~[hibernate-core-5.6.15.Final.jar:5.6.15.Final]
        ... 11 more
2023-05-11T06:04:56,044 ERROR o.h.engine.jdbc.spi.SqlExceptionHelper [quartzScheduler_Worker-8] Connection is closed
2023-05-11T06:06:05,947 INFO  o.s.scheduling.quartz.LocalDataSourceJobStore [QuartzScheduler_quartzScheduler-NON_CLUSTERED_MisfireHandler] Handling 1 trigger(s) that missed their scheduled fire-time.
2023-05-11T06:07:34,931 INFO  c.z.reporting.schedule.LoadModelJob [quartzScheduler_Worker-1] Updated batch state  extractor=6 type=MODEL batch=831 state=LOADING

Incorrect credentials

Error messages similar to the following can appear in the ETL log file when the credentials on the REPORTS > Zenoss Analytics page are incorrect.

2023-06-01T08:25:23,644 ERROR c.z.reporting.schedule.ExtractPerfJob [quartzScheduler_Worker-4] [HttpResponseException] status code: 401, reason phrase: Unauthorized>
2023-06-01T08:25:23,644 DEBUG c.z.reporting.extract.service.PerfExtractor [perfextractjob-7] Extraction began  batch=244 alias="usedfilesystemspace__bytes" task="usedfilesystemspace__bytes-1426"
2023-06-01T08:25:23,644 ERROR c.z.reporting.extract.service.PerfExtractionTask [perfextractjob-10] Failed to execute request: status code: 401, reason phrase: Unauthorized  batch=244 alias="UPDATE.updateHandler.commits.1m" task="UPDATE.updateHandler.commits.1m-1429" exception="class org.apache.http.client.HttpResponseException"
org.apache.http.client.HttpResponseException: status code: 401, reason phrase: Unauthorized
2023-06-01T08:25:23,644 ERROR c.z.reporting.extract.service.PerfExtractionTask [perfextractjob-4] Failed to execute request: status code: 401, reason phrase: Unauthorized  batch=244 alias="UPDATE.updateHandler.cumulative" task="UPDATE.updateHandler.cumulative-1423" exception="class org.apache.http.client.HttpResponseException"