# Effective pagination over composite aggregation in Java **URL:** https://discuss.elastic.co/t/effective-pagination-over-composite-aggregation-in-java/226638 **Category:** Elasticsearch **Created:** [April 6, 2020, 7:30am UTC](https://discuss.elastic.co/t/effective-pagination-over-composite-aggregation-in-java/226638 "2020-04-06T07:30:49Z") **Posts on this page:** 4 **Page:** 1
### Author: ![florinmoc](https://avatars.discourse-cdn.com/v4/letter/f/dbc845/32.png) [@florinmoc](https://discuss.elastic.co/u/florinmoc) #### Post date: [April 6, 2020, 7:30am UTC](https://discuss.elastic.co/t/effective-pagination-over-composite-aggregation-in-java/226638/1 "2020-04-06T07:30:49Z")
Hi, I'm trying to paginate over composite aggregation in Java. I have some working code, but it doesn't seem to be very efficient, as it re-builds the query each time. Is there a way to re-use the query and just change the "afterKey" parameter? Here is the code that I tried: ```auto public static void main(String[] args) throws IOException { Instant before = Instant.now(); RestHighLevelClient client = new RestHighLevelClient( RestClient.builder(new HttpHost("localhost", 9200, "http"), new HttpHost("localhost", 9201, "http"))); int searchAfterKey = -1; System.out.println("After key start: " + searchAfterKey); Map responseMap = null; boolean emptyBuckets = false; while (!emptyBuckets) { responseMap = getProfiles(client, searchAfterKey); emptyBuckets = (Boolean) responseMap.get("emptyBuckets"); if (!emptyBuckets) { searchAfterKey = (Integer) responseMap.get("afterKeyVal"); } } System.out.println("After key end: " + searchAfterKey); client.close(); Instant after = Instant.now(); long delta = Duration.between(before, after).toMillis(); System.out.println("Time taken: " + delta + " ms."); } private static Map getProfiles(RestHighLevelClient client, int searchAfterKey) throws IOException { SearchSourceBuilder sourceBuilder = new SearchSourceBuilder(); sourceBuilder.size(0); QueryBuilder rangeQuery = QueryBuilders.rangeQuery("effectiveDate").from(0).to(1575518400, false); QueryBuilder boolQuery = QueryBuilders.boolQuery().must(rangeQuery); ConstantScoreQueryBuilder constScoreQB = new ConstantScoreQueryBuilder(boolQuery); List> sources = new ArrayList>(); TermsValuesSourceBuilder id = new TermsValuesSourceBuilder("agg_on_id").field("id"); sources.add(id); CompositeAggregationBuilder compositeAggregationBuilder = new CompositeAggregationBuilder("by_id", sources); Map afterKey = new HashMap(); afterKey.put("agg_on_id", searchAfterKey); compositeAggregationBuilder.size(10000).aggregateAfter(afterKey).subAggregation(AggregationBuilders .topHits("latest_snapshot").sort("effectiveDate", SortOrder.DESC).size(100).fetchSource(true)); sourceBuilder.query(constScoreQB).aggregation(compositeAggregationBuilder); SearchRequest searchRequest = new SearchRequest("from_dynamo"); searchRequest.source(sourceBuilder); System.out.println(searchRequest.source().toString()); // response SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT); ParsedComposite aggs = searchResponse.getAggregations().get("by_id"); for (ParsedComposite.ParsedBucket bucket : aggs.getBuckets()) { for (Aggregation pth : bucket.getAggregations().asList()) { SearchHit[] searchHit = ((ParsedTopHits) pth).getHits().getHits(); for (SearchHit sh : searchHit) { System.out.println(sh.getSourceAsString()); } } } boolean emptyBuckets = aggs.getBuckets().isEmpty(); Map responseMap = new HashMap(); if (!emptyBuckets) { Integer afterKeyVal = (Integer) aggs.afterKey().get("agg_on_id"); responseMap.put("afterKeyVal", afterKeyVal); } responseMap.put("emptyBuckets", emptyBuckets); return responseMap; } ``` Thanks, Florin ---
### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet) #### Post date: [April 6, 2020, 7:43am UTC](https://discuss.elastic.co/t/effective-pagination-over-composite-aggregation-in-java/226638/2 "2020-04-06T07:43:25Z")
Welcome ! Not sure it's a problem. But anyway, have a look at [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html) ---
### Author: ![florinmoc](https://avatars.discourse-cdn.com/v4/letter/f/dbc845/32.png) [@florinmoc](https://discuss.elastic.co/u/florinmoc) #### Post date: [April 7, 2020, 6:22am UTC](https://discuss.elastic.co/t/effective-pagination-over-composite-aggregation-in-java/226638/3 "2020-04-07T06:22:41Z")
Thanks David for your quick answer! Templates could be a way forward, but I seem unable to implement them in Java. I've written this code (using RestHighLevelClient as I understand TransportClient is deprecated) ```auto public static void main(String[] args) throws IOException { Instant before = Instant.now(); RestHighLevelClient client = new RestHighLevelClient( RestClient.builder(new HttpHost("localhost", 9200, "http"), new HttpHost("localhost", 9201, "http"))); Map template_params = new HashMap<>(); template_params.put("agg_on_id", -1); SearchResponse sr = new SearchTemplateRequestBuilder((ElasticsearchClient) client) .setScript("{\n" + " \"size\": 0,\n" + " \"query\": {\n" + " \"constant_score\": {\n" + " \"filter\": {\n" + " \"bool\": {\n" + " \"must\": [\n" + " {\n" + " \"range\": {\n" + " \"effectiveDate\": {\n" + " \"from\": 0,\n" + " \"to\": 1575518400,\n" + " \"include_lower\": true,\n" + " \"include_upper\": false,\n" + " \"boost\": 1.0\n" + " }\n" + " }\n" + " }\n" + " ],\n" + " \"adjust_pure_negative\": true,\n" + " \"boost\": 1.0\n" + " }\n" + " },\n" + " \"boost\": 1.0\n" + " }\n" + " },\n" + " \"aggregations\": {\n" + " \"by_id\": {\n" + " \"composite\": {\n" + " \"size\": 10000,\n" + " \"sources\": [\n" + " {\n" + " \"agg_on_id\": {\n" + " \"terms\": {\n" + " \"field\": \"id\",\n" + " \"missing_bucket\": false,\n" + " \"order\": \"asc\"\n" + " }\n" + " }\n" + " }\n" + " ],\n" + " \"after\": {\n" + " \"agg_on_id\": {{agg_on_id}}\n" + " }\n" + " },\n" + " \"aggregations\": {\n" + " \"latest_snapshot\": {\n" + " \"top_hits\": {\n" + " \"from\": 0,\n" + " \"size\": 1,\n" + " \"version\": false,\n" + " \"seq_no_primary_term\": false,\n" + " \"explain\": false,\n" + " \"_source\": {\n" + " \"includes\": [],\n" + " \"excludes\": []\n" + " },\n" + " \"sort\": [\n" + " {\n" + " \"effectiveDate\": {\n" + " \"order\": \"desc\"\n" + " }\n" + " }\n" + " ]\n" + " }\n" + " }\n" + " }\n" + " }\n" + " }\n" + "}") .setScriptType(ScriptType.INLINE) .setScriptParams(template_params) .setRequest(new SearchRequest("from_dynamo")) .get() .getResponse(); client.close(); Instant after = Instant.now(); long delta = Duration.between(before, after).toMillis(); System.out.println("Time taken: " + delta + " ms."); } ``` But when i try to run this code, I get this error: ```auto Exception in thread "main" java.lang.ClassCastException: class org.elasticsearch.client.RestHighLevelClient cannot be cast to class org.elasticsearch.client.ElasticsearchClient (org.elasticsearch.client.RestHighLevelClient and org.elasticsearch.client.ElasticsearchClient are in unnamed module of loader 'app') at elasticClient.ElasticRequestCreator.main(ElasticRequestCreator.java:27) ``` Could you suggest a way out? Florin ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [May 5, 2020, 6:22am UTC](https://discuss.elastic.co/t/effective-pagination-over-composite-aggregation-in-java/226638/4 "2020-05-05T06:22:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.