Our ES is getting slower and slower day by day. I have read about
different people using ES, and our usage is very less compared to what
people have posted.
I am sure we are doing something wrong with our design. Could any one
please suggest us some improvements.
We have the following data
- Log data (5 million documents per day)
- Our primary db data.
a) Type A - 100,000 documents (growth 5% per month)
b) Type B - 500,000 documents (growth 5% per month)
c) Type C - 30 Million Documents (growth 15% per month)
I would prefer to have these under one index
- Metrics events (around 100 k documents per day)
On a big EC2 instance(2x large), we have created ES with 2 shards(both
on same machine)
We wanted to use parent child relation ship. So we created an index,
and made logs, metrics, type B, type C documents as child's to type
A(accounts). Things were fine until we started loading log data. But
now after we loaded around 200 Million of those logs, every thing is
slow. Parent child queries even for types with less data take long
long time making it unusable for us.
The performance of individual queries also decreased a lot (still
Now we are struck. we are not sure if we need to make any changes in
the design so that we can use parent child queries with improved
efficiency, or should we move our log type(or even more) in to a
different index so that things will be fast?
Any suggestions are greatly appreciated.
Thank you very much for your time reading my post.