Documents not getting udpated as per the pipeline

I am using one elapsed plugin to calculate the time difference between two events, with the output sending data into a "merged" index. For some reason, at certain times, the merged index is not getting updated in real time, and because of the delay, wrong alerts are being triggered.
I cant seem to figure out the reason behind the same, as its happening quite randomly.

I am sharing the filter section of my pipeline below, the input contains data ingestion from 3 kafka topics, and the output section is sending data to ES, with every document being updated against a fingerprint.

filter {
    json {
        source => "message"
    }

    fingerprint {
        source => ["[data][args][conversationId]"]
    }
    
    if [data][args][chatData][source] == "zd:agentWorkspace" {
        mutate {
            add_tag => [ "taskStarted" ]
        }
    }
    if [data][args][chatData][source] == "web" {
        mutate {
            add_tag => [ "taskTerminated" ]
        }
    }

    elapsed {
        start_tag => "taskStarted"
        end_tag => "taskTerminated"
        unique_id_field => "fingerprint"
        timeout => 190
        keep_start_event => "last"
    }
    if "elapsed_expired_error" in [tags] {
        mutate {
            remove_field => [ "[host]" ]
        }
    }
}

This is a trimmed version of the filter section. I have removed the mutate and elasticsearch filter plugins in there.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.