Transform on fields of a nested object

Elastic Stack Elasticsearch
1

Does transform support group by on nested fields?

I was trying to group by on a field of a nested object but no results appear in the preview.

2

Hi,

transform should support nested fields, I am not aware of a limitation. Your issue sounds like a bug. Can you post your configuration and some example data?

Did you try it in dev console? Did you try to create the transform despite preview not showing something?

What version are you on?

3

Hi Hendrik,

ES Version: 7.7.1
I tried using both KIbana and APIs. The generated index is empty.

Please see the mapping below. I was trying to join on the annotations.theme field.

FYI - This join field is optional hence not every document will have this

Mapping

        {
      "test2" : {
        "mappings" : {
          "properties" : {
            "_class" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "annotations" : {
              "type" : "nested",
              "properties" : {
                "sentiment" : {
                  "type" : "integer"
                },
                "theme" : {
                  "type" : "keyword"
                }
              }
            },
            "comment" : {
              "type" : "text"
            },
            "datetime" : {
              "type" : "date"
            },
            "externalId" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
            },
            "source" : {
              "type" : "keyword"
            },
            "title" : {
              "type" : "text"
            }
          }
        }
      }
    }
4

Thanks, I misunderstood the question. I understood nested as in "nested object" but not "nested data type".

Aggregations on nested fields require the nested aggregation, so this is not a transform limitation, but a limitation in the aggregation framework. Transform is using composite aggregations, which does not support this case.

I doubt that we add support for it, because this has scalability issues. Un-nesting is limited but composite aggregations and transform are meant for bigger data sets.

Maybe you can change you input data layout or if the data is small just use an ordinary aggregation instead of a transform.

5

Thanks! We are seeing similar issue in using the ML module i.e anomaly detection on a nested field.

Is the same limitation applicable there as well?

6

Can you open a new thread and elaborate a bit (config, sample data)?
Are you using aggregations in the datafeed?

(Having that said, I guess you have a platinum license, so it might be better to go via support)

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.