Monitoring API value is updated only after logstash restart

[dandrestor (Dan Nestor)]

Hi there

I am trying to monitor the number of events that are currently pending in the persisted queue of my Logstash instance. This number seems to update only when I restart Logstash. I looked at the metric in three different places, they all agree:

• The monitoring UI in Kibana (see screenshot below),
• The contents of the .monitoring indexes, specifically the queue.events_count field, and
• The monitoring API endpoint via curl, specifically the value pipeline.queue.events.

The value only changes immediately following a restart of the logstash daemon, and then stays constant:

I confirmed the value remained constant even as events from the queue were replayed through to the back-end ES cluster following a network interruption. Am I doing anything wrong, or do I have the wrong expectation?

I am running Logstash 5.5.2 on Linux. The contents of my logstash.yml file:

path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.url: [
  "http://host1:9200",
  "http://host2:9200",
  "http://etc:9200"
]
queue.type: persisted
path.queue: "/path/to/queue"
queue.max_bytes: "200gb"
http.host: "x.x.x.x"
http.port: 9600

Thanks,
Dan

[jordansissel]

I tested this on Logstash 5.6.1:

% grep -v '^#' config/logstash.yml
queue.type: persisted

% bin/logstash -e 'input { generator { } } filter { sleep { time => 0.010 } }'

% while true; do ; date | tr '\n' ' ' ; curl -s localhost:9600/_node/stats/pipeline?pretty | jq .pipeline.queue.events; sleep 1; done
Fri Oct  6 13:51:38 PDT 2017 2302534
Fri Oct  6 13:51:39 PDT 2017 2539903
Fri Oct  6 13:51:40 PDT 2017 2539903
Fri Oct  6 13:51:41 PDT 2017 2539903
Fri Oct  6 13:51:42 PDT 2017 2539903
Fri Oct  6 13:51:43 PDT 2017 2539903
Fri Oct  6 13:51:44 PDT 2017 2776786
Fri Oct  6 13:51:45 PDT 2017 2776786
Fri Oct  6 13:51:46 PDT 2017 2776786
Fri Oct  6 13:51:47 PDT 2017 2776786
Fri Oct  6 13:51:48 PDT 2017 3017269
Fri Oct  6 13:51:49 PDT 2017 3017269
Fri Oct  6 13:51:50 PDT 2017 3017269
Fri Oct  6 13:51:51 PDT 2017 3017269
Fri Oct  6 13:51:52 PDT 2017 3017269
Fri Oct  6 13:51:53 PDT 2017 3244089

Demonstrated above, the counter is incrementing over time.

It may be an oddity that the counter value updates once every 5-ish seconds, but it certainly is updating for me.

[dandrestor (Dan Nestor)]

Perhaps it's been fixed in the meantime? I will try to upgrade to and test again.