Queries with large character counts in fields

[jmb13562 (Jason)]

Hi all,

I've run into an issue where searching through a relatively small data set is hitting some pretty slow performance. We're running queries for text matches like "Person Name" against a dataset of around 8GB. The problem is that this data is ingested documents like word, pdf's, or excel docs. Some of the content fields have 16+ million characters in them. Does anybody have advice on how to handle fields with such a large character count?

Any help or advice is much appreciated!
Thanks,
Jason

[cjcenizal]

Hi Jason, thanks for posting your question! Text fields should be tokenized by default which will optimize search performance to the point where 16M characters in a text field shouldn't cause a problem.

Would you mind sharing one of the queries that's slow so we can get a better idea of what's happening? Could you please also share your mapping for the index you're searching?

Thanks,
CJ

[jmb13562 (Jason)]

Thanks for replying!

Here is the query that is being issued.

https://gist.github.com/jmb13562/f8f61dcf91fda92ac0f85bb7a657c43b

Here are the mappings of the index. Some searches will result in query times upwards of 7-10 minutes.

https://gist.github.com/jmb13562/294a40060a835c7ec0a5001a714eb8de

[cjcenizal]

Thanks Jason! I believe the highlighter is the culprit here. Using highlighter with large fields will always incur performance considerations because it has to load the entire text, analyze it, and search it.

Here's a thread from a user with a similar problem as yours: Highlighting takes long time for large documents.

Here are some solutions to consider:

• Set index_options to offsets in the mapping. You'll have to reindex your data to try this solution. This will allow the default unified highlighter to skip the analysis step. See https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html#offsets-strategy for more info.
• Set term_vector to with_position_offsets in the mapping. This also requires reindexing. It will also increase the size of the index significantly. See the above links for more info on this option.
• Use the fast vector highlighter by setting your highlighter type to fvh. Use this option in conjunction with the term_vector option. Comes with the same tradeoff. See https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html#fast-vector-highlighter.

[jmb13562 (Jason)]

Thanks CJ!

So if I'm understanding this correctly then I'd need to set these options per field within the mappings?

[jmb13562 (Jason)]

Answered my own question there. Thanks again for the help!