Java API and Search Template with params

Elastic Stack Elasticsearch
1

I have a template based on https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html, it looks like this for one param, which is meant to be an in query

            "          {\n" +
            "            \"terms\": {\n" +
            "              \"assignedToOrg\": [\"{{#orgIds}}\",\n" +
            "          \"{{.}}\",\n" +
            "          \"{{/orgIds}}\"]" +
            "            }\n" +
            "          },\n" +

I'm pass a list param for the param orgIds..

    List<Integer> orgIds = Arrays.asList(2,3);
    long from = 1426452221000l;
    long to = 1434401021000l;
    Map<String,Object> params = new ImmutableMap.Builder<String,Object>()
            .put("orgIds", orgIds).put("from",from).put("to",to).build();

For some reason, the rendered query ends up looking like this

   {"terms":{"organizationId":["","2","","3",""]}}]}

Which gives back a number format exception. How do I get rid of the extra entries here?

2

I think it might be that you are adding a ' before and after each element. This stack overflow answer seems to suggest you can do the following to convert your array to a comma delimited string (I haven't tried this myself though):

{
    "terms": {
        "assignedToOrg": [
            {{#orgIds}}
                "{{.}}"{{^last}}, {{/last}}
            {{/orgIds}}
        ]
    }
}

This will add each element in the array and then append a comma unless it's the last element in the array

HTH

3

Sorry, I misread the stackoverflow comment and thought {{^last}} was a special identifier, when actually the user had just added last = true as a property to the last object. I think the cause of your problem is as I mentioned above but I haven't yet found an elegant solution to it.

Sorry

4

Actually I missed the simple solution which is to just pass the array into the template directly rather than trying to iterate over the contents to build the array. So your script could be the following:

{
    "terms": {
        "assignedToOrg": {{orgIds}}
    }
}

Hope this helps (and you can follow my slightly erratic messages :slight_smile: )

5

I've tried what you have listed before. It doesn't seem to work, at least when using the Java API to upload a template. It complains about invalid JSON

                    "            {\n" +
                    "              \"terms\": {\n" +
                    "                \"organizationId\": {{orgIds}}" +
                    "              }\n" +
                    "            }\n" +

Which using {{orgIds}} is invalid JSON. It seems like some escaping is required to make that work. Stack is below, line 29 is the {{orgIds}} line.

Caused by: org.elasticsearch.common.jackson.core.JsonParseException: Unexpected character ('{' (code 123)): was expecting either valid name character (for unquoted name) or double-quote (for quoted) to start field name
at [Source: UNKNOWN; line: 29, column: 37]
at org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1419)
at org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508)
at org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:437)
at org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._handleOddName(UTF8StreamJsonParser.java:1808)
at org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1496)
at org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:693)

6

Hmm yes you are right. You'll need to provide the search template as either a file template or an escaped JSON string. For the latter, the following should hopefully work (note that the below example is the entire search template):

{ \"query\": { \"terms\": { \"organisationId\": {{orgIds}} } } }

So in JAVA code that would then need to be further escaped to:

{ \\\"query\\\": { \"terms\\\": { \\\"organisationId\\\": {{orgIds}} } } }

See if that does the trick.

If you want to use a file template instead have a look at: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html#pre-registered-templates

7

I don't see how double escaping everything fixes it. The error is on the {{ characters in orgIds. The template works fine when I put in the hardcoded array instead of the parameter, e.g.

                    "              \"terms\": {\n" +
                    "                \"organizationId\": [1,2]" +
                    "              }\n" +

Pre-registered templates aren't a solution for me since I'm using an embedded ES. I'd prefer to not deploy anything direct to the server.

8

The escaping helps because when Elasticsearch is parsing the search template request (before running the mustache engine) it needs valid JSON. The {{orgIds}} is not valid JSON (since you cannot have an object within an object without specifying a field name). By passing the template in as an escaped JSON string, the request parser can parse the template as a String instead of a JSON Object. It can then pass this string template through the mustache engine to replace the mustache placeholders to get the final JSON request which can then be run.

Your hardcoded example works because it does not contain a mustache placeholder and is valid JSON, so the request parser can successfully parse it.

The note box above the heading I linked you to in my previous post explains this with an example using conditional mustache clauses. The principle here is the same.

9

I just tried, in both Java and REST API. Issue remains. In fact, with the REST API I get the same parse error.

10

Ok, I think I understand now.

  1. The JSON has to be flattened.
  2. You're storing the template as a string. Because of that, the template itself is wrapped in "", making it not JSON in the body, and the overall document just being {"template":"some string"}
  3. the {{#orgIds}}.. solution doesn't work, even with this pattern. The correct way to do it is via the {{orgIds}}

Thanks for your patience!