Improve search performance on a specific index

[satishkumar (satish)]

Elastic search version: 2.1
I have an index with below configuration

"number_of_shards":5
"number_of_replicas":2
i.e primary shards 5 and replica per each primary as 2.

Number of nodes in cluser: 3

I have deployed each node on a single machine with 32gb ram and 3 machines to form a cluster.

For elastic process, configured -Xms15G and -XmX15G

I have indexed around 70k documents and pri.store size is 9gb and the store size i.e complete index size is 27gb.

When i search with this configuration, when i search for documents with some criteria on this index, Elastic search takes around 6-9 seconds on an average

Other indices are workfin fine with a searh performance of below 500ms but this index is giving issues.

What are possible options to improve the peformance?

1. what should be maximum size of a single document to have optimal search pefromance.
2. should the index be split into multiple indices for better perfromance? what should be max index size or no of documents in a index for optimal search performance

[polyfractal (Zachary Tong)]

The size of the index is very small (70k documents), so it's likely something about your query that's causing the slowdown. Could you post your query? Also, if you run it with profile: true we can look at the profile timing results which may be helpful

[satishkumar (satish)]

Thanks Tong for the reply. I have checked the query and "highlighting" is causing the issue.

There are some documents in the index where single document is having 7-14 mb of text.

fast vector highlighter is used with below configuration.
"store": true
"term_vector" : "with_positions_offsets"

Does highlighting pose any issue with large documents of 7-14mb text?. In some of the documents, a single field is around 5mb.

Are there any recommendations for the size of single json document?