Elasticsearch Performance Problem

[blanks]

Hi, I hope Im in the right place for this.
I currently working with elasticsearch for log analysis, my source are IIS logs. I encounter a very slow indexing rate with logstash and elasticsearch. I think I can rule out logstash as problem, since I ran logstash only with stdout and there it was very fast.
My setup ist a single windows computer with 16gb run. One elasticsearch node and kibana running in docker and logstash runs directly on the system. I get arroung 100-300 index/s and I think thats very slow. Is this normal for this setup ? I attached a few statistics.

[polyfractal (Zachary Tong)]

have you tried increasing the flush size of Logstash? It may be that the bulk size being sent is too small for your system.

The 100% "Active Time" for that drive looks poor however. What kind of drive is it? Is it heavily fragmented? A slow drive would definitely slow down the entire pipeline and lead to poor indexing performance.

[blanks]

Thanks for the answer, the drive is a normal 7200 rpm drive. There is only my windows backup on it and the data for elasticsearch.
Are there any recommended values for the flush size in logstash, the docs just say there is no default value?

[polyfractal (Zachary Tong)]

Flush size depends on the size of the documents being indexed. E.g. one hundred 50byte docs is very different from one hundred 10kb docs.

I'd just start increasing it step-wise until you find performance stops improving, then you've likely found a good place. Usually a few hundred to few thousand docs per bulk is acceptable, depending on size.

[blanks]

I played arround with flush size, but it didnt make a diffrence. So I changed the harddrive of elasticsearch to my ssd, now I get arround 1100 / s . Is this expected or should it be faster ?
Currently I have one Index per Month and 5 Shards per Index.

Thanks for your help