Ruby code to iterate through list of pipelines received by Logstash API

Elastic Stack Logstash
1

Hi, I am trying to implement monitoring and alerting for some of our pipelines that we have set up in logstash. My thought is to run a cron job on the Logstash API (specifically _nodes/stats/pipelines) to retrieve both the amount of events being emitted by each pipeline, and also the size of the persisted queues for each pipeline.

I know how to use ruby code to iterate through an array of items, but am unfamiliar with how to accomplish this with the way the logstash API formats the response.

    "pipelines": {
        "pipeline1": {
            "events": {
                "in": 0,
                "filtered": 0,
                "out": 0,
                "duration_in_millis": 0,
                "queue_push_duration_in_millis": 0
            },
            "queue": {
                "type": "persisted",
                "events_count": 0,
                "queue_size_in_bytes": 1,
                "max_queue_size_in_bytes": 26843545600
            }
      }
       "pipeline2": {
            "events": {
                "in": 0,
                "filtered": 0,
                "out": 0,
                "duration_in_millis": 0,
                "queue_push_duration_in_millis": 0
            },
            "queue": {
                "type": "persisted",
                "events_count": 0,
                "queue_size_in_bytes": 1,
                "max_queue_size_in_bytes": 26843545600
            },
        }
    }

Removing all the unnecessary information, presuming there are many more objects that pipeline1 and pipeline2, I want to know how to iterate over each of these objects so that I can check the values of queue.queue_size_in_bytes and events.out to ensure everything is running as expected. I was unfortunately unable to find any sort of documentation or examples on how to accomplish this and was hoping someone might have some idea or be able to point me in the right direction.

Thanks in advance!

2

If you have valid JSON (yours is not) in a field called pipielines, then this

    ruby {
        code => '
            p = event.get("pipelines")
            p.each { |k, v|
                puts k + " " + v["events"]["out"].to_s + " " + v["queue"]["queue_size_in_bytes"].to_s
            }
        '
    }

will print

pipeline1 0 1
pipeline2 0 1

Obviously you will have to add code that tests whether "everything is running as expected".

3

Thanks! I presume I probably made an error when shortening the json down to just the important parts, but yes the JSON should be valid when we ingest it from the API. I will give this a shot, thank you.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.