PutIndexTemplateRequest and custom plugin

Hello,
About the Java client, I have the same problem as

when I try to use the similarity that I defined in my plugin.

I can create the index template using PUT with curl.
But I have the same exception as #299 if I try to use the java client.
I have Elastic and co.elastic.clients version 8.17.0.

If I do

"similarity": {
        "my_similarity": {
          "type": "my_similarity_type"
        }

where my_similarity_type is defined in my plugin, it works with curl, but I get the same exception when I use the client with the same code as #299

co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch.indices.SettingsSimilarity: Unknown 'type' value: 'my_similarity_type' (JSON path: settings.similarity.nemo) (in object at line no=9, column no=18, offset=232)

note: everything works when I don't use my plugin and when I do

"similarity" : {
                "my_bm25" : {
                  "type" : "BM25",
                  "b" : "0.0",
                  "k1" : "0.0"
                }
              }

Bonjour Antoine :wink:

Could you share the full stack trace please?
Also your java code would help.

May be there's a way to provide the type using something like Kind("my_similarity_type") but I'm unsure. @ltrotta would be able to tell more.

Worse case, you can fallback to the low level client to solve the immediate problem.

Hello!
I have to investigate a bit more because I was pretty sure the problem was already solved, but it seems like the solution was reversed at some point. I'll get back ASAP.

actually, #299 is fixed, but the similar problem I have, is when the similarity type comes from a plugin. To me, it is another ticket.

As for the Java and the trace, here is what I have:

elastic-simple.json

{
  "index_patterns": [
    "test_similarity"
  ],
  "template": {
    "settings": {
      "index": {
        "similarity": {
          "my_similarity": {
            "type": "nemo-custom"
          }
        }
      }
    }
  }
}

elastic-simple-bm25.json

{
  "index_patterns": [
    "test_similarity"
  ],
  "template": {
    "settings": {
      "index": {
        "similarity": {
          "my_bm25": {
            "type": "BM25",
            "b": 0.0,
            "k1": 0.0,
            "discount_overlaps": true
          }
        }
      }
    }
  }
}

Here is what I use to reproduce the problem with Elastic 8.17.0 in a container. It works with elastic-simple-bm25.json but doesn't with elastic-simple.json:

        private void discussElasticSimple() {
            String jsonString;
            try {
                jsonString = EsYesUtils.readResource("elastic-simple.json");
            } catch (IOException e) {
                throw new RuntimeException(e);
            }

            InputStream stream = new ByteArrayInputStream(jsonString.getBytes(StandardCharsets.UTF_8));
            PutIndexTemplateRequest putIndexTemplateRequest = new PutIndexTemplateRequest.Builder()
                    .name("my-template")
                    .create(true)
                    .withJson(stream)
                    .build();

            System.out.println("template = " + putIndexTemplateRequest.template());
}

Here is the trace:

Caused by: co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch.indices.SettingsSimilarity: Unknown 'type' value: 'nemo-custom' (JSON path: template.settings.index.similarity.my_similarity) (in object at line no=9, column no=29, offset=154)
	at co.elastic.clients.json.ObjectDeserializer.lambda$setTypeProperty$0(ObjectDeserializer.java:359)
	at co.elastic.clients.json.ObjectDeserializer.parseUnknownField(ObjectDeserializer.java:284)
	at co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:229)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:157)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77)
	at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:47)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:349)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:333)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78)
	at co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:214)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:157)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78)
	at co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:214)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:157)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78)
	at co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:214)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:157)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78)
	at co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:214)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166)
	at co.elastic.clients.util.WithJsonObjectBuilderBase.withJson(WithJsonObjectBuilderBase.java:54)
	at co.elastic.clients.json.WithJson.withJson(WithJson.java:43)

It works when I use curl:

curl -X PUT -H 'Content-Type: application/json' -u elastic:s3cret! -k -d @/Users/lef/dev/esyes-configuration/src/test/resources/elastic-simple.json http://127.0.0.1:33043/_index_template/search-index-template

Hello! I can confirm that this is the exact same issue, which was fixed and accidentally reversed shortly after :sweat_smile: a new fix is ready and it will be merged shortly, hopefully permanently this time. It should be part of the next patch, so 8.17.2 :slight_smile:

OK thank you,
Is it related to elasticsearch-specification again ?
When was the regression introduced ? Could I test with a former version ?

Unfortunately the fix was never released, because of bad timing: it should've been out with 8.13.0, but then the regression was introduced just before the final build. And yes it's still the commonelasticsearch-specification, all clients are generated from it. 8.17.2 should be out around next week, if it takes longer then I'll point you to a snapshot version of it that you will be able to test.

1 Like

Hello,
I confirm that 8.17.2 of the java client solves the problem,
thank you