Hourly aggregation for timestamp and Java API irrespective of date

[jpmiitian (Jay Prakash Meena)]

Hi

Though we can get hourly buckets using script as described in below link.
But, Can I get the same result using Elasticsearch Java Rest HL client.
Is it possible to define the script for AggregationBuilder.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html

POST /sales/_search?size=0
{
    "aggs": {
        "dayOfWeek": {
            "terms": {
                "script": {
                    "lang": "painless",
                    "source": "doc['date'].value.dayOfWeek"
                }
            }
        }
    }
}

Please provide your suggestions.

Thanks
Jay

[colings86 (Colin Goodheart-Smithe)]

Yes, the TermsAggregationBuilder has a .script(Script) method you can pass your script to.

[jpmiitian (Jay Prakash Meena)]

Thanks for responding Colin.

I was using the .script(Script) method of TermsAggregationBuilder.
But there is an error coming while searching the elasticsearch.

Please refer below code block.

public Map findByHour(String pixelId, Long startTime, Long endTime) {
    BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
    boolQueryBuilder.must(QueryBuilders.matchQuery("pid", pixelId));

    QueryBuilder queryBuilder = boolQueryBuilder.filter(QueryBuilders.rangeQuery("eventTime").gte(startTime).lt(endTime));

    // date histogram

// DateHistogramAggregationBuilder dateHistoAggBuilder =
// AggregationBuilders.dateHistogram("date_histogram").field("eventTime")
// .dateHistogramInterval(DateHistogramInterval.DAY)
// .format("dd-MM-yyyy");

    // aggs for day of the week
    // Script dayOfWeekScript =  new Script("doc['eventTime'].date.dayOfWeek");

    //    inline = new Script("doc['eventTime'].date.dayOfWeek");

    Script inline = new Script(ScriptType.INLINE, "painless", "doc['eventTime'].date.dayOfWeek", new HashMap<>());

    TermsAggregationBuilder termsAggBuilderForDOW = AggregationBuilders.terms("by_day_of_week").script(inline);
    termsAggBuilderForDOW.size(7); // 7 size

    List<AggregationBuilder> aggregationBuilderList = new ArrayList<>();

// aggregationBuilderList.add(dateHistoAggBuilder);
aggregationBuilderList.add(termsAggBuilderForDOW);

    SearchRequest searchRequest = new SearchRequest("esIndex"); // index name
    searchRequest.types("indexType");// type name
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    searchSourceBuilder.query(queryBuilder);

    aggregationBuilderList.stream().forEach(termsAggregationBuilder -> {
        searchSourceBuilder.aggregation(termsAggregationBuilder);
    });

    searchSourceBuilder.fetchSource(false);//don't fetch docs.
    searchRequest.source(searchSourceBuilder);//set search source builder


    // make search request for client
    try {
        // getting error here
        SearchResponse searchResponse = esServiceIndex.searchViaRestHLClient(searchRequest);
        Double totalHits = (double) searchResponse.getHits().getTotalHits();
        if (totalHits == 0L) {
            return null;
        }
        Map responseMap = new HashMap();

        Aggregations aggregations = searchResponse.getAggregations().get("date_histogram");


        Terms weekdayTerms = searchResponse.getAggregations().get("by_day_of_week");

        // weekday map
        weekdayTerms.getBuckets().stream().forEach(weekdayBucket -> {
            responseMap.put(weekdayBucket.getKeyAsString(), weekdayBucket.getDocCount());
        });

        return responseMap;

    } catch (Exception e) {
        log.error("Error in searching.", e);
    }
    return null;
}

//Error logs.

error":{"root_cause":[{"type":"illegal_argument_exception","reason":"[script] unknown field [source], parser not found"}],"type":"illegal_argument_exception","reason":"[script] unknown field [source], parser not found"},"status":400}
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:354) ~[rest-5.5.3.jar:5.6.2]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:343) ~[rest-5.5.3.jar:5.6.2]
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) ~[httpcore-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) ~[httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) ~[httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) ~[httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.5.jar:4.4.5]

Regards
Jay

[cbuescher (Christoph)]

@jpmiitian trying to reproduce, just to make sure I'm looking at the right thing: which HL client version are you using exactly and which version is the elasticsearch server that you are querying?

[jpmiitian (Jay Prakash Meena)]

Hi @cbuescher
HL version is 5.6.2 and ES is 5

[cbuescher (Christoph)]

Which ES version exactly?

The problem is probably that the 5.6.2 client renders the Script class as the following Json:

{
	"source": "doc['eventTime'].date.dayOfWeek",
	"lang": "painless"
}

But older versions of Elasticsearch (I think everything older than 5.5) can only parse:

{
	"inline": "doc['eventTime'].date.dayOfWeek",
	"lang": "painless"
}

The renaming of inline to script was done in https://github.com/elastic/elasticsearch/pull/25127 and both parameter names are supported in 5.6 and the 6.x.

Your backend should be on at least 5.6. so this can work. In general, the HL client is supposed to be forward compatible, while there may be bw comp issues even for non breaking changes made to ES core.

[jpmiitian (Jay Prakash Meena)]

Thanks @cbuescher for informing regarding the compatibility.

The ES version is 5.5,
I noted that below was working on kibana console.
{ "inline": "doc['eventTime'].date.dayOfWeek", "lang": "painless" }

[cbuescher (Christoph)]

Yes, it works as long as you specify the query manually and use "inline" instead of "source". The 5.6 client uses "source" when rendering the REST request though, since this is the preferred variant of the parameter name starting with 5.6.