Insert Events with Changed Status Only using Logstash

[sajid (Sajid Ansari)]

Hi All,

Need your help to solve an issue. Below are the details:

Pipeline1 : Polls multiple http endpoints every minute using http_poller and insert the response to index1 in ES.

Below is the sample response of one of the endpoints:

{
    "status": "Healthy",
    "totalDuration": "00:00:00.0052662",
    "entries": {
        "testDep-1": {
            "data": {},
            "duration": "00:00:00.0010479",
            "status": "Healthy"
        },
        "testDep-2": {
            "data": {},
            "duration": "00:00:00.0018730",
            "status": "Healthy"
        },
        "testDep-3": {
            "data": {},
            "duration": "00:00:00.0023060",
            "status": "Healthy"
        }
    }
}

This logstash pipeline split the response of a http endpoint to multiple events ex. for above it creates 1+3 events. Refer screenshot below:

Pipeline2 : Query index1 using elasticsearch input plugin with {"range": {"@timestamp": {"gte": "now-60s","lte": "now"}}} so that only last response of http_poller can be picked. Then transform some data in filter section and insert into index2 as shown in below image:

The Pipeline2 is inserting all the events from index1 to index2. Now the requirement is we don't want all the events in index2. We only need events when "status" or "dependency.status" gets changed. i.e. if a URL is responding with "unhealthy" and once that response comes back to healthy state. You can understand it as we need alerts only in the index2.(When a http endpoints becomes unhealthy and when it comes to healthy again without repetitive events ).

We are using basic license and can't use watcher. We want to do this using logstash only. Please guide how can we achieve this.

[Badger]

You could use an elasticsearch filter to query index1 for the current value of those fields on the most recent document and drop{} the event if they are unchanged. Alternatively you could use ruby and cache the most recent value for those fields and event.cancel if they are unchanged.

[sajid (Sajid Ansari)]

Hi @Badger,

I tried elasticsearch filter plugin but not couldn't get the desired result. If possible can you please help with some sample code?