Hey All,
I am doing the stress test on Elasticsearch where I'm continuously creating indexes using single thread, maximum of 2000 index. But the index creation time has slowed down as the index count increases, all operations from java client is taking long time to respond and also sometime client is timing out. Does index creation cause any performance issues?
How many shards are you creating per index?
Total 5! As per default settings
That's probably why. 5 shards for 2000 indices = 10000 total indices, way to omany.
So I am collecting millions of record every day and performing aggregation every 30 min on some fields for past 24 hours. In current one index approach, elasitc is becoming too slow. So I am trying horizontal scaling to create multiple index. What is the good approach? One index per hour or day?
Millions a day is not much, why not just use one daily index?
Does it really matter if I create 4 indexes per day or one index per day where there is 500 to 700 million record? Will it improves the performance when there is an aggressive aggregation happen every minute? Our current bottleneck is repeatedly we run into OOM within in one or two days. Even if we give 25GB of a heap to elastic.
That's getting out of the scope of your original question, might be better to keep that in the other thread.
Sure. Thank you