Hi I have some questions based on pagination for reading an index.
(by the way, FIFO doesn't need to be guranteed, just need a bulk reading process with speed)
obviously in java using the HighRestClient, I have heard that this isn't thread safe thus
my intentions to create a multithreaded pagination reading code was dismissed....
recently, I have found a function called msearch which might slightly speed the process due to
the reduce of connection latency for a request.
By using a code that repeats scrolling and searching, is it possible to use msearch in such procedure?
final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(30L));
SearchRequest searchRequest = new SearchRequest();
searchRequest.scroll(scroll);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery()
.must(QueryBuilders.matchQuery("collectDate", today).operator(Operator.AND))
.must(QueryBuilders.matchQuery("collectHour", hour).operator(Operator.AND))
.must(QueryBuilders.matchQuery("type", type).operator(Operator.AND))
.must(QueryBuilders.matchQuery("useAttr", useAttr).operator(Operator.AND));
searchSourceBuilder.query(queryBuilder);
searchSourceBuilder.fetchSource(true);
searchSourceBuilder.fetchSource(include, exclude);
searchSourceBuilder.size(5000);
searchRequest.indices(esIndexName);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
String scrollId = searchResponse.getScrollId();
SearchHit[] searchHits = searchResponse.getHits().getHits();
while (searchHits != null && searchHits.length > 0) {
for (int i = 0; i < searchHits.length; i++) {
///////////////////////////////service code that adds each docs to a List object in the java HEAP ///////////////////////////////////////////
}
SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
scrollRequest.scroll(scroll);
searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
scrollId = searchResponse.getScrollId();
searchHits = searchResponse.getHits().getHits();