Hi,
I use elasticsearch 7.2 on a single node computer.
Computer features:
- 64 gb ram
- 2 tb hdd (the disk which ES installed is 500 gb)
- 8 cores cpu
- Win10 OS
My ES configurations&features
-106 indices
-106 shards
-0 replica
-~150 gb total size
-32 gb jvm heap size
The features of data I try to index
-Most of them have almost 100 fields
-A few of them have 1000-2500 fields (I increased max length of field property)
- All rows are numeric(long,int,double etc)
- I don't use mapping, elasticsearch makes it automatically when first document is indexing.
-My document batch size (rows per request) is 2000. I tried between 500-15k and 2k document per request look like optimum.
I make a javascript code for indexing(it sends data with required json format by using elasticsearch.js library). At the begining, my indexing rate is between 2.5-4 mb/s. The rate is almost constant until size of indiceses reach 100-120 gb. After that, the rate of indexing has been dramaticaly decreased. Until 100-120 gb, ES can respond my requests with a really few latency, but after 120 gb size, it responds almost one time in every 10 minutes.
I try too many things to fix this situation (including the advices in your performance tunning guide) but nothing is changed.
I think my main problem is jvm heap size. Because I've realised that in kibana monitoring tool, even I don't make any indexing (or send any request) jvm heap size usage value is about 18gb/32 gb, and when I start indexing, it easly reachs 32 gb (in-circuit-breaker error occurs) in just half an hour. Because my js script keep sending request consecutively but even just 1 request takes 10 minutes for executing by elasticsearch. So, the size of requests in queue(threadpool) become enormous.
I don't understand the behaivour of jvm. In the current situation I have just 1 shard with 30 gb, a few of shards are 10-15 gb and the most of them are 4-5 gb. All shards are initialized and green.
I try to many things for understanding and solving this situation but have not achieve yet.
Can anyone explain this?
Why elasticsearch in my computer uses %50 of total memory when the size of indexed data just %5-6 of my total harddisk.
Why its indexing rate decreased so dramatically after 100-120 gb indexed data (1 execution per 8-10 minutes)?
What are possible solutions?
Thank you.