Rightsizing elastic batch size and number of workers

[alaine (alex)]

I am trying to right size my configs for the logstashes that I have feeding my elastic stack. They are pushing filebeat and winlogbeat data. Right now they seem to be fairly underutilized and I am seeing a little bit of ingest lag within the stack.

Right now my servers have 8 cpus and 15gb of memory. We have configured logstash to use 6 workers, a pipeline.batch.size of 1000 and a pipeline.batch.delay of 150. I have been reading through the documentation and haven't been able to come up with a decent formula for right sizing the situation. I don't have a problem playing with the numbers, but figured I would ask here in case someone has some valuable advice.

Thanks

[ahmed_charafouddine]

In the event that your data sent by the beats (filebeat, metricbeat, etc.) passes through logstash, be sure to assign the value of the log date to @timestamp.

[warkolm (Mark Walkom)]

I don't know what that comment has to do with the question @ahmed_charafouddine?

[warkolm (Mark Walkom)]

There is no formula because it's a bit of a complex question.

The rough figure is to aim for is ~5MB of data, or <5000 documents/events in each bulk request.

[alaine (alex)]

@ahmed_charafouddine thanks for the answer, but I am confused.

@warkolm so I am seeing rougly 3200events/sec on the logstash that I have been talking about. So it would seem like I would just set pipeline.batch.size to 3500? Though I have access to 6 cpus. So should that batch size be smaller like 500 if I make pipeline workers 6?