ElasticSearch Cluster JVM Heap Increasing level while doing Queries and Writting

Elastic Stack Elasticsearch
1

Hello,
I have three Indexes, one have 11 million documents, I need to do 35.000 searches in that index. Is there a way to that and not increase my JVM Heap to the point where elastic fails? I'm using the Scroll API, and making queries that have size of 5000.

I also get this error:

org.elasticsearch.client.ResponseException: method [POST], host , URI [/ensinosuperior/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&search_type=query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [8127961922/7.5gb], which is larger than the limit of [8127315968/7.5gb], real usage: [8127961440/7.5gb], new bytes reserved: [482/482b]","bytes_wanted":8127961922,"bytes_limit":8127315968,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [8127961922/7.5gb], which is larger than the limit of [8127315968/7.5gb], real usage: [8127961440/7.5gb], new bytes reserved: [482/482b]","bytes_wanted":8127961922,"bytes_limit":8127315968,"durability":"PERMANENT"},"status":429}

2

can you explain why you need to use the scroll API for 35k requests? Also, are those concurrent? If so, the scroll API is not the way to go and you may want to look at the search_after feature.

--Alex

3

Basically, each request queries one field in these 11 million documents, and totalizes its values, so I need the scroll API to scroll in these 11 million docs.

I'll check this search_after feature, thanks.

4

Can’t you use an aggregation for that?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.