Logstash [7.8] problem with an elastic update doc

Elastic Stack Logstash
1

Hi Guys,

Running into a bit of an issue that I am trying to overcome.

I have a index that only updates documents, specifically I am monitoring devices that are either in an UP or DOWN status and the doc_id is the name of the device. The issue i have is that when there is a DOWN / UP in really quick succession it can't workout which is the latest status as it seems to bulk post to elasticsearch

Initially, i thought that the 4 pipeline workers was causing the problem and transferred the doc's i need to update to a new pipeline with a single worker. This still didn't fix the issue so i set the pipeline.batch.delay,

pipeline.batch.delay: 2

Still having the issue with the batch post not knowing what the latest status is.

Can anyone provide any work around for this? I was thinking of maybe trying the aggregate plugin to solve this issue.

2

If you want to avoid the batching that the elasticsearch output does you could use an http filter to send the data to elasticsearch.

3

Thanks @Badger I haven't used the http filter to send to elasticsearch before.

I'll investigate and see how it goes.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.