AMQ anti stuck monitoring

There is a process that watches AMQ and checks if any queue is stuck for too long. Sends notification email to OPS if such queue is found. Currently applies to LH, A3 and EW.

Process goes through AMQ logs to filter out messages (separate process writes in logs the state of queues every minute). It then compares numbers within each specific queue and, if there was no increase in dequeued messages during the last X minutes (threshold configurable, default is 5min) then queue is likely stuck.

When notification is sent, stamp file is created. File contains number of dequeued messages and file name follows pattern of \${curProcess.channel}_\${curProcess.process}.queue

If there is a stamp file, it is compared to newer processed log entries. If file-content dequeued number is lower than in new logs, it means the queue has processed something and therefore another notification email is sent, resolving the issue.

Solving the conflict

Main point of this monitoring is to make sure queues aren't stuck for too long. There is no unified solution and it's case-by-case. Some may be processing large amounts of data and taking long time to finish. Others may get stuck and need to be investigated. Possibly no consumer for whatever reason.