Rightsizing elastic batch size and number of workers

Elastic Stack Logstash
1

I am trying to right size my configs for the logstashes that I have feeding my elastic stack. They are pushing filebeat and winlogbeat data. Right now they seem to be fairly underutilized and I am seeing a little bit of ingest lag within the stack.

Right now my servers have 8 cpus and 15gb of memory. We have configured logstash to use 6 workers, a pipeline.batch.size of 1000 and a pipeline.batch.delay of 150. I have been reading through the documentation and haven't been able to come up with a decent formula for right sizing the situation. I don't have a problem playing with the numbers, but figured I would ask here in case someone has some valuable advice.

Thanks

2

In the event that your data sent by the beats (filebeat, metricbeat, etc.) passes through logstash, be sure to assign the value of the log date to @timestamp.

3

I don't know what that comment has to do with the question @ahmed_charafouddine?

4

There is no formula because it's a bit of a complex question.

The rough figure is to aim for is ~5MB of data, or <5000 documents/events in each bulk request.

5

@ahmed_charafouddine thanks for the answer, but I am confused.

@warkolm so I am seeing rougly 3200events/sec on the logstash that I have been talking about. So it would seem like I would just set pipeline.batch.size to 3500? Though I have access to 6 cpus. So should that batch size be smaller like 500 if I make pipeline workers 6?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.