Search is very slow on an index with 150million documents


(Kiran Madabhushi) #1

Hi,
I am just starting elastic search and I have created an index per day with each index storing around 150million documents.
Each document is a log line from a test run. The log line has a field for the test run which is a non-analyzed url.
When I do a filtered query on the index for this field, it takes more than 40 seconds to give me the result. What am I doing wrong? How can I speed this up?

Here is an example format of the log line

filename:line

All the lines has the test-url.

Earlier I indexed each test run into its own index (test-url1 test-url2 as indexes after replacing the special chars). I was creating more than 400 indexes per day and it was not scaling after 15 days. But the search was very fast.

I have 4 nodes in my cluster and each index has 2 shards. Will making it 4 shards help?


(system) #2