Logstash pipeline monitoring

[funlearning321 (Learning Fun)]

Hello,

Can i please know if there is a way to monitor the logstash pipelines that can trigger an alert if the pipeline is down using ELK? we have a X-pack.

Thanks

[ylasri (Yassine LASRI)]

you can use Logstash monitoring API with Watcher

[funlearning321 (Learning Fun)]

Thank you for the reply. I can use node stats API from the logstash monitoring API. But i dont see any attribute from pipeline stats that can help to understand whether the pipeline up or down. below is the sample pipeline stats outptut.

{
"pipelines" : {
"test" : {
"events" : {
"duration_in_millis" : 365495,
"in" : 216485,
"filtered" : 216485,
"out" : 216485,
"queue_push_duration_in_millis" : 342466
},
"plugins" : {
"inputs" : [ {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-1",
"events" : {
"out" : 216485,
"queue_push_duration_in_millis" : 342466
},
"name" : "beats"
} ],
"filters" : [ {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-2",
"events" : {
"duration_in_millis" : 55969,
"in" : 216485,
"out" : 216485
},
"failures" : 216485,
"patterns_per_field" : {
"message" : 1
},
"name" : "grok"
}, {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-3",
"events" : {
"duration_in_millis" : 3326,
"in" : 216485,
"out" : 216485
},
"name" : "geoip"
} ],
"outputs" : [ {
"id" : "35131f351e2dc5ed13ee04265a8a5a1f95292165-4",
"events" : {
"duration_in_millis" : 278557,
"in" : 216485,
"out" : 216485
},
"name" : "elasticsearch"
} ]
},
"reloads" : {
"last_error" : null,
"successes" : 0,
"last_success_timestamp" : null,
"last_failure_timestamp" : null,
"failures" : 0
},
"queue" : {
"type" : "memory"
}
},
"test2" : {
"events" : {
"duration_in_millis" : 2222229,
"in" : 87247,
"filtered" : 87247,
"out" : 87247,
"queue_push_duration_in_millis" : 1532
},
"plugins" : {
"inputs" : [ {
"id" : "d7ea8941c0fc48ac58f89c84a9da482107472b82-1",
"events" : {
"out" : 87247,
"queue_push_duration_in_millis" : 1532
},
"name" : "twitter"
} ],
"filters" : ,
"outputs" : [ {
"id" : "d7ea8941c0fc48ac58f89c84a9da482107472b82-2",
"events" : {
"duration_in_millis" : 139545,
"in" : 87247,
"out" : 87247
},
"name" : "elasticsearch"
} ]
},
"reloads" : {
"last_error" : null,
"successes" : 0,
"last_success_timestamp" : null,
"last_failure_timestamp" : null,
"failures" : 0
},
"queue" : {
"type" : "memory"
}
}
}