Logstash [7.8] problem with an elastic update doc

Hi Guys,

Running into a bit of an issue that I am trying to overcome.

I have a index that only updates documents, specifically I am monitoring devices that are either in an UP or DOWN status and the doc_id is the name of the device. The issue i have is that when there is a DOWN / UP in really quick succession it can't workout which is the latest status as it seems to bulk post to elasticsearch

Initially, i thought that the 4 pipeline workers was causing the problem and transferred the doc's i need to update to a new pipeline with a single worker. This still didn't fix the issue so i set the pipeline.batch.delay,

pipeline.batch.delay: 2

Still having the issue with the batch post not knowing what the latest status is.

Can anyone provide any work around for this? I was thinking of maybe trying the aggregate plugin to solve this issue.

If you want to avoid the batching that the elasticsearch output does you could use an http filter to send the data to elasticsearch.

Thanks @Badger I haven't used the http filter to send to elasticsearch before.

I'll investigate and see how it goes.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.