Hello. I'm using Elasticsearch 8.1, dependency co.elastic.clients Elasticsearch-java version 8.1.2. When creating one scroll request for 15_000 records, a large memory consumption occurs. The request consumed 8 GB of memory and vened 8 GB after 20 - 25 minutes (1 GB freed up after about 6 minutes). Who's to say what's the matter? The code for the scroll request is given below:
RestClient httpClient = RestClient.builder(new HttpHost("localhost", 9200)).build();
ElasticsearchTransport transport = new RestClientTransport(httpClient, new JacksonJsonpMapper());
ElasticsearchClient client = new ElasticsearchClient(transport);
public Long createScrollRequest(String input) throws IOException {
//List<News> list = new LinkedList<>();
SearchRequest searchRequest = new SearchRequest.Builder()
.scroll(new Time.Builder().time("20s").build())
.query(qb -> qb.multiMatch(mmqb -> mmqb.query(input).fields("status")))
.build();
SearchResponse<News> searchResponse = client.search(searchRequest, News.class);
String scrollId = searchResponse.scrollId();
List<Hit<News>> searchHits = searchResponse.hits().hits();
while (searchHits != null && searchHits.size() > 0) {
ScrollRequest scrollRequest = new ScrollRequest.Builder()
.scroll(new Time.Builder().time("20s").build())
.scrollId(scrollId)
.build();
searchResponse = client.scroll(scrollRequest, News.class);
scrollId = searchResponse.scrollId();
searchHits = searchResponse.hits().hits();
}
ClearScrollRequest clearScrollRequest = new ClearScrollRequest.Builder()
.scrollId(scrollId)
.build();
client.clearScroll(clearScrollRequest);
return searchResponse.hits().total().value();
}
If we add property searchRequest.size (10000) it saves memory. Instead of Gigabytes, 100 - 300 Megabytes will be spent. Which is quite a lot (