I am using one elapsed plugin to calculate the time difference between two events, with the output sending data into a "merged" index. For some reason, at certain times, the merged index is not getting updated in real time, and because of the delay, wrong alerts are being triggered.
I cant seem to figure out the reason behind the same, as its happening quite randomly.
I am sharing the filter section of my pipeline below, the input contains data ingestion from 3 kafka topics, and the output section is sending data to ES, with every document being updated against a fingerprint.
filter {
json {
source => "message"
}
fingerprint {
source => ["[data][args][conversationId]"]
}
if [data][args][chatData][source] == "zd:agentWorkspace" {
mutate {
add_tag => [ "taskStarted" ]
}
}
if [data][args][chatData][source] == "web" {
mutate {
add_tag => [ "taskTerminated" ]
}
}
elapsed {
start_tag => "taskStarted"
end_tag => "taskTerminated"
unique_id_field => "fingerprint"
timeout => 190
keep_start_event => "last"
}
if "elapsed_expired_error" in [tags] {
mutate {
remove_field => [ "[host]" ]
}
}
}
This is a trimmed version of the filter section. I have removed the mutate and elasticsearch filter plugins in there.