I am trying to right size my configs for the logstashes that I have feeding my elastic stack. They are pushing filebeat and winlogbeat data. Right now they seem to be fairly underutilized and I am seeing a little bit of ingest lag within the stack.
Right now my servers have 8 cpus and 15gb of memory. We have configured logstash to use 6 workers, a pipeline.batch.size of 1000 and a pipeline.batch.delay of 150. I have been reading through the documentation and haven't been able to come up with a decent formula for right sizing the situation. I don't have a problem playing with the numbers, but figured I would ask here in case someone has some valuable advice.
In the event that your data sent by the beats (filebeat, metricbeat, etc.) passes through logstash, be sure to assign the value of the log date to @timestamp.
I don't know what that comment has to do with the question @ahmed_charafouddine?
There is no formula because it's a bit of a complex question.
The rough figure is to aim for is ~5MB of data, or <5000 documents/events in each bulk request.
@ahmed_charafouddine thanks for the answer, but I am confused.
@warkolm so I am seeing rougly 3200events/sec on the logstash that I have been talking about. So it would seem like I would just set pipeline.batch.size to 3500? Though I have access to 6 cpus. So should that batch size be smaller like 500 if I make pipeline workers 6?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.