Logstash 2.0 Monitoring API?


(Nate Fox) #1

I kept seeing mentions of a monitoring API when LS 2.0 hit, but cant seem to find any mention of it in the current docs. Did it get implemented?


(Magnus Bäck) #2

No, it wasn't implemented. See the issue below and the Logstash roadmap.


(Nate Fox) #3

Bummer. Looks like it got pushed out to LS 3.0? I'm guessing there's not even an ETA for that version.

Is there a way to see how many lines are queued up and waiting to move out of LS? I've been using rsyslog and rely on its ability to provide queue stats but I'd like to move to something a bit more powerful.


(Magnus Bäck) #4

Queued up where? Logstash has no internal queue to speak of.


(Nate Fox) #5

If Logstash cant write to its destination, it would in theory be queued from the filter to output, right? I'm basing this knowledge off of the roadmap verbiage here: https://www.elastic.co/guide/en/logstash-roadmap/current/index.html#_resiliency

Basically, I want to know if logstash is behind on sending a fast filling file to its destination for whatever reason (usually because the destination cant handle the ingestion rate).


(Magnus Bäck) #6

If Logstash cant write to its destination, it would in theory be queued from the filter to output, right?

Yes and no. There are two 20-event buffers in the pipeline but that's all there is in terms of queueing. In practice a clogged output will stall the whole pipeline.

Basically, I want to know if logstash is behind on sending a fast filling file to its destination for whatever reason (usually because the destination cant handle the ingestion rate).

Logstash won't give you such metrics, but since file inputs are observable from the outside you could compare the sincedb offset to the file size and see how far behind Logstash is. See http://stackoverflow.com/a/33146636/414355.


(Nate Fox) #7

Thanks for that stack overflow link. I might be able to cobble something together using that.

Do you know if these are the kinds of metrics that are planned with the monitoring API?


(Magnus Bäck) #8

Do you know if these are the kinds of metrics that are planned with the monitoring API?

I don't know. Have you checked the GitHub issues?


(Mark Walkom) #9

I don't have a date for 3.X, but it won't be far off.


(system) #10