v7 v8

Pipeline monitoring

The pipelined status web service gives information on the status of the pipelined process.

It can be accessed manually using a browser or automatically with a monitoring application.

It’s in REST format, which is described below.

Indicators

This section lists the indicators in the status web service.

Recommended indicators to monitor are highlighted.

• Consumer: name of the client pulling the triggers. Configured in the pipeline option.
• http-request
  • last-alive-ms-ago: time in ms since a connection check was made.
  • last-failed-cnx-ms-ago: time in ms since the last time the connection check failed.
  • pipeline-host: name of the host where the pipeline data is pulled from.
• pointer
  • current-offsets: value of the pointer into the pipeline, per child thread.
  • last-flush-ms-ago: time in ms since a batch of triggers was retrieved.
  • next-offsets-flush: time to wait until the next batch, when finished.
  • processed-since-last-flush: number of triggers processed in the last batch.
• routing
  • triggers: list of triggers retrieved. Configured in the pipelined option.
• stats
  • average-pointer-flush-time-ms: average processing time for one batch of triggers.
  • average-trigger-processing-time-ms: average time spent parsing the triggers data.
  • bytes-read: number of bytes read from the queue since the process was started.
  • current-messages: current number of pending messages that have been pulled from the queue and are awaiting processing. This indicator should be close to zero.
  • current-retries: current number of messages that have failed processing and are awaiting retry.
  • peak-messages: maximum number of pending messages the process has been handling since it was started.
  • pointer-flushes: number of batches of messages processed since the start.
  • routing-JS-custom: number of messages that were processed by the custom JS.
  • trigger-discarded: number of messages that were discarded after too many retries due to processing errors.
  • trigger-processed: number of messages that were processed without an error.
  • trigger-received: number of messages received from the queue.

These stats are displayed per processing thread.

• average-trigger-processing-time-ms: average time spent parsing the triggers data.

• is-JS-processor: value “1” if this thread uses the custom JS.

• trigger-discarded: number of messages that were discarded after too many retries due to processing errors. This indicator should be zero.

• trigger-failures: number of processing errors in the JS. This indicator should be zero.

• trigger-received: number of messages received from the queue.

• Settings: they are set in the config files.

  • flush-pointer-msg-count: number of messages in a batch.
  • flush-pointer-period-ms: time between two batches, in milliseconds.
  • processing-threads-JS: number of processing threads running the custom JS.
  • retry-period-ms: time between two retries when a processing error occurs.
  • retry-validity-duration-ms: duration from the time processing is retried until the message is discarded.
  • Pipeline messages Report

Pipeline messages report

This report displays the number of messages per hour in the last five days.