GetTransformResponse unable to parse "source" field

Craig_Uss · November 4, 2020, 4:34pm

ELK 7.8.1
Using Springboot 2.2.5
I am able to create a transform using the Kibana Stack Management and using the PUT _transform curl command. However when trying to retrieve the same transform using the High Level REST client, I receive the following error:

Caused by: org.elasticsearch.common.xcontent.XContentParseException: [1:137] [transform_config_source] failed to parse field [query]
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:531) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:541) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:324) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:171) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:163) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.client.transform.transforms.TransformConfig.lambda$static$1(TransformConfig.java:99) ~[elasticsearch-rest-high-level-client-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareObject$1(AbstractObjectParser.java:169) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:386) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:529) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	... 47 common frames omitted
Caused by: org.elasticsearch.common.xcontent.XContentParseException: unknown named object category [org.elasticsearch.index.query.QueryBuilder]
	at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:128) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:385) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:320) ~[elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.client.transform.transforms.QueryConfig.fromXContent(QueryConfig.java:39) ~[elasticsearch-rest-high-level-client-7.8.1.jar:7.8.1]
	at org.elasticsearch.client.transform.transforms.SourceConfig.lambda$static$1(SourceConfig.java:56) ~[elasticsearch-rest-high-level-client-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareObject$1(AbstractObjectParser.java:169) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:386) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:529) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]
	... 55 common frames omitted

dadoonet · November 4, 2020, 4:43pm

Welcome!

What are the dependencies you declared for your project?

Asking because when using springboot with elasticsearch, you need to be explicit with some transitive dependencies as SpringBoot declares a version 6.4...

Basically you can put this in your pom.xml:

<properties>
  <elasticsearch.version>7.9.3<elasticsearch.version>
</properties>

See documentation here: https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto-customize-dependency-versions

Craig_Uss · November 4, 2020, 4:44pm

Thanks for the response.

Yes right now have it declared as
<elasticsearch.version>7.8.1</elasticsearch.version>

dadoonet · November 4, 2020, 4:45pm

Could you share a bit of your code?

Craig_Uss · November 4, 2020, 4:49pm

Sure. Here is the transform that I created:

{
  "id": "bogus",
  "source": {
    "index": [
      "*bogus*"
    ],
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "exists": {
                    "field": "items.id"
                  }
                }
              ],
              "minimum_should_match": 1
            }
          },
          {
            "bool": {
              "should": [
                {
                  "exists": {
                    "field": "counter.type"
                  }
                }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "index": "bogus"
  },
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "5m"
    }
  },
  "pivot": {
    "group_by": {
      "items.id.keyword": {
        "terms": {
          "field": "items.id.keyword"
        }
      },
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1d"
        }
      }
    },
    "aggregations": {
      "unique_item_investigations": {
        "filter": {
          "bool": {
            "must": [
              {
                "term": {
                  "counter.type": "test"
                }
              },
              {
                "term": {
                  "event.metadata.double_click": false
                }
              }
            ]
          }
        },
        "aggs": {
          "cardinality_unique_item_investigations": {
            "cardinality": {
              "field": "counter.type.keyword"
            }
          }
        }
      }
    }
  },
  "description": "Bogus",
  "settings": {}
}

Here is the java snippet used to call and retrieve the transform

GetTransformRequest getTransformRequest = new GetTransformRequest(transformName);
        try {
            GetTransformResponse getTransformResponse = client.transform().getTransform(getTransformRequest, RequestOptions.DEFAULT);
            return getTransformResponse.getCount() > 0;
        } catch (ElasticsearchException e) {
            logger.info(String.format("Transform - %s - not found", transformName));
        }
        return false;

dadoonet · November 4, 2020, 5:22pm

Great thanks. I can reproduce the problem...

I opened

Craig_Uss · November 4, 2020, 6:34pm

Perfect, thank you so much!

BenTrent · November 5, 2020, 7:52pm

@Craig_Uss what are you supplying as your NamedXContent serializers in your high level rest client constructor?

The error

unknown named object category [org.elasticsearch.index.query.QueryBuilder]

Indicates that for some reason the appropriate named xcontent serializers were not provided to the RestHighLevelClient object.

Please provide your RestHighLevelClient version information and how you are calling the constructor. That will help narrow this down.

BenTrent · November 5, 2020, 8:02pm

Digging around in the code, it seems that

static List<NamedXContentRegistry.Entry> getProvidedNamedXContents() {
        List<NamedXContentRegistry.Entry> entries = new ArrayList<>();
        for (NamedXContentProvider service : ServiceLoader.load(NamedXContentProvider.class)) {
            entries.addAll(service.getNamedXContentParsers());
        }
        return entries;
    }

In a 7.8 high level rest client loads all the xcontent parsers for the included plugins. If the search plugin is not included, it won't be loaded I think.

EDIT:

Here is a way to construct a client that includes the SearchModule xcontent serializers.

class HighLevelClient extends RestHighLevelClient {
       HighLevelClient(RestClient restClient) {
            super(restClient, RestClient::close, new SearchModule(Settings.EMPTY, Collections.emptyList()).getNamedXContents());
        }
    }

Craig_Uss · November 6, 2020, 6:56pm

class HighLevelClient extends RestHighLevelClient {
       HighLevelClient(RestClient restClient) {
            super(restClient, RestClient::close, new SearchModule(Settings.EMPTY, Collections.emptyList()).getNamedXContents());
        }
    }

This worked out for me! Thank you so much for your help on this!

dadoonet · November 6, 2020, 7:18pm

Great. But I do think that end users should not have to care about that.
It should be available OOTB.

Craig_Uss · November 6, 2020, 7:24pm

I agree. What I am having trouble with reconciling is building a Transform. In other API operations, I usually have the option to provide the Json source to build the object. Here I have to create a xContentParser. Something like the below (yet to get it to work, but just to give you an idea)

XContentParser parser = JsonXContent.jsonXContent.createParser(new NamedXContentRegistry((new TransformNamedXContentProvider()).getNamedXContentParsers()), LoggingDeprecationHandler.INSTANCE, transformSource);
        TransformConfig transformConfig = TransformConfig.fromXContent(parser);

Is this the right approach for trying to create a PutTransformRequest? I can open a new thread if needed!

system · December 4, 2020, 7:24pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.