Query Template not working

Elastic Stack Elasticsearch
1

Hi,

I have created a query template, which involves multiple aggregations with default fields as below:

{ "template" : {
          "size": 0,
          "aggs": {
            "range": {
              "date_range": {
                "field": "{{time_field}}{{^time_field}}@timestamp{{/time_field}}",
                "format": "yyyy-MM-dd",
                "ranges": [
                  {
                    "from": "{{start_date}}{{^start_date}}now-1w/w{{/start_date}}",
                    "to": "{{end_date}}{{^end_date}}now/w{{/end_date}}"
                  }
                ]
              },
              "aggs": {
                "agg_by_interval": {
                  "date_histogram": {
                    "field": "{{time_field}}{{^time_field}}@timestamp{{/time_field}}",
                    "format": "yyyy-MM-dd HH:mm:ss",
                    "interval": "{{interval}}{{^interval}}1w{{/interval}}"
                  },
                  "aggs": {
                    "groupby_field": {
                      "terms": {
                        "size": 0,
                        "field": "{{group_by_field}}"
                      },
                      "aggs": {
                        "percentiles_of_field": {
                          "percentiles": {
                            "field": "{{analytics_field}}",
                            "percents": ["{{#percents}}", "{{.}}", "{{/percents}}"]
			}
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
}

When I run the template, it simply retrieves the first 10 documents and does not perform any aggregations.
Whereas when I run the same query without using the template it works fine.

Am I missing something with the templates here?
I have run other simpler templates which work fine for me.

Thanks.

2

I get a feeling that templates are not working with POST based searches. It is anything to do with that? If so how to overcome that.

Please help.

3

Hi,
doesn't seem like the reason could be POST vs GET, how did you get to this conclusion? That said I am not sure yet what goes wrong with your request, nothing I can see at first glance.

The render search template endpoint would be great to use here, see https://github.com/elastic/elasticsearch/issues/6821 , but it comes with 2.0 beta 1 only. Maybe you can give it a try though.

4

Thanks @javanna
Regarding POST vs GET:
I am not entirely sure why but, as we know there are some searches that work with only POST and others with GET.
The above query when I run with GET it get error. And with POST I get the correct result.
But whereas when I use template to run the same query, I get error with POST, and GET does not give correct result. [Actually returns records without any aggregations]

So I wonder, how the query template is executed at the back end to handle POST and GET based queries.

5

Can you please post the error that you get back when you use GET?

6

Sorry @javanna , I do not get error with GET. The query returns the documents but not aggregated results. Similar to the result when I use GET with template.

I get the following error with POST using the template:

Query

{
  "template": {
    "id": "PERCENTILES_OF_FIELD_IN_DATE_RANGE"
  },
  "params": {
    "start_time": "2015-08-17||-1w/w",
    "end_time": "2015-08-17||+1w/w",
    "group_by_field": "apiname-short",
    "analytics_field": "response-time (ms)",
    "percents" : [ 90, 99 ]
  }
}

Error

"error": "SearchPhaseExecutionException[Failed
 to execute phase [query], all shards failed; shardFailures 
{[b_iKE2bJStC-yX1HRFfvow][logstash-enabler-2015.07.01][0]: 
SearchParseException[[logstash-enabler-2015.07.01][0]: 
from[-1],size[-1]: Parse Failure [Failed to parse source 
[{"aggs":{"range":{"aggs":{"agg_by_interval":{"aggs":{"groupby_field":{"aggs":{"percentiles_of_field":{"percentiles":{"field":"response-time
 
(ms)","percents":["","90","","99",""]}}},"terms":{"field":"apiname-short","size":0}}},"date_histogram":{"field":"@timestamp","interval":"1w","format":"yyyy-MM-dd
 
HH:mm:ss"}}},"date_range":{"field":"@timestamp","ranges":[{"to":"now/w","from":"now-1w/w"}],"format":"yyyy-MM-dd"}}},"size":0}]]];
 nested: NumberFormatException[empty String]; }
...

, "status": 400
}
8

Hi @javanna
Can you please look into the error posted above.
Still trying to figure out what is wrong here.

Thanks.

9

This is what is causing the error. The percents passed into the percentiles agg need to be numbers and "" is not a number

10

Yes @colings86
I observed the same. But the "" are inserted by template.
I tried with input as:

 "percents" : [99]

Still it inserts "" and generates this:

"percents":["","99",""]
11

FYI I am using Elasticsearch 1.5.2

12

Please let me know if this issue is identified and resolved?

13

I don't really have a good solution for you, this is more like a "me too" type of reply.

Anyway I also ran into this issue a while ago and I just couldn't get the template to work. It seemed that the mustache template parser always inserts those quotes. My issue was with document ids. When I tried to make a query that searched for something based on a list of ids I couldn't get it to work. A single id worked just fine, but when I tried to specify a list of anything I always got the quotes. I noticed that the same template works if I put it in a file in the scripts folder in the node. I lived with that for a while, but eventually I got frustrated with how awkward it is to maintain the template files and I got rid of them. So now I have given up on templates and I simply have my own templating system based on constant strings that I fill with required details. It's far from ideal but what can you do...

So the template thing would be great if it worked, but as it stands it seems broken.

14

Without having dug too deeply into the template query you posted this sounds a lot like the following issue:

Can you check you are still having this issue with ES 2.0?