# Ingest processor failure when accessing nested field

**URL:** <https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194>\
**Category:** Elasticsearch\
**Created:** [August 13, 2018, 3:02pm UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194 "2018-08-13T15:02:25Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![bukem](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@bukem](https://discuss.elastic.co/u/bukem)\
**Post date:** [August 13, 2018, 3:02pm UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/1 "2018-08-13T15:02:26Z")

</div>

Hello,  
I'm facing the following problem: accessing nested field within ingest processor fails with an error. Simple example on how to reproduce the problem:

1. ` curl -X PUT "localhost:9200/nest_test" -H 'Content-Type: application/json' -d'
{
"mappings":{
"resources":{
"properties":{
"nest":{
"type":"nested",
"dynamic":"strict",
"properties":{
"field_raw":{ "type":"keyword" },
"field_lower":{ "type":"keyword" }
}
}
}
}
}
}
'
`

2. ` curl -X PUT "localhost:9200/_ingest/pipeline/nested_lowercase" -H 'Content-Type: application/json' -d'
{
"description" : "failure reproducing",
"processors" : [
{
"lowercase" : {
"field": "nest.field_raw",
"target_field": "nest.field_lower"
}
}
]
}
'
`

3. ` curl -X POST "localhost:9200/nest_test/1?pretty&refresh&pipeline=nested_lowercase" -H 'Content-Type: application/json' -d'
{"nest": [{"field_raw": "Test Me"}]}
'
`

There similar topic with the same problem, but different processors, but no answers anywhere ☹

[https://discuss.elastic.co/t/nested-field-not-getting-processed-by-ingest-processor-plugin/96136](https://discuss.elastic.co/t/nested-field-not-getting-processed-by-ingest-processor-plugin/96136)  
[https://discuss.elastic.co/t/nested-documents-in-ingest-pipeline-processor/135305](https://discuss.elastic.co/t/nested-documents-in-ingest-pipeline-processor/135305)  
[https://github.com/elastic/elasticsearch/issues/22193](https://github.com/elastic/elasticsearch/issues/22193)

Is there a way to access and work with nested fields from processors?

Thanks in advance for your help,  
Mike

---

<div class="post-metadata">

**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\
**Post date:** [August 13, 2018, 6:13pm UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/2 "2018-08-13T18:13:10Z")

</div>

I think you will need to use a script processor. None of the processors know how to deal with a nested field (which appears as an array of sub-maps). Eg, for doing lowercase with your example:

```auto
"processors" : [
    {
        "script" : {
            "source": "for (Map nested : ctx._source['nest']) { nested['field_lower'] = nested['field_raw'].toLowerCase(Locale.ROOT); }"
        }
    }
]

```

---

<div class="post-metadata">

**Author:** ![talevy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talevy/32/44896_2.png) [@talevy](https://discuss.elastic.co/u/talevy)\
**Post date:** [August 13, 2018, 6:29pm UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/3 "2018-08-13T18:29:58Z")

</div>

Another way to achieve this is with the help of the `foreach` processor.

here is an example in Console

```
PUT _ingest/pipeline/nest_lowercase
{
  "description": "lowercase field_raw",
  "processors": [
    {
      "foreach": {
        "field": "nest",
        "processor": {
          "lowercase": {
            "field": "_ingest._value.field_raw",
            "target_field": "_ingest._value.field_lower"
          }
        }
      }
    }]
}

PUT nest_test/_doc/1?pipeline=nest_lowercase
{
  "nest": [{"field_raw": "CamelCase"}]
}

GET nest_test/_doc/1

```

The document that was indexed, will look like this:

```
{
  "_index": "nest_test",
  "_type": "_doc",
  "_id": "1",
  "_version": 1,
  "found": true,
  "_source": {
    "nest": [
      {
        "field_lower": "camelcase",
        "field_raw": "CamelCase"
      }
    ]
  }
}

```

Since `foreach` only accepts one processor at a time to operate on the array elements, you may find it more customizable to use Painless scripting for additional transformations

---

<div class="post-metadata">

**Author:** ![bukem](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@bukem](https://discuss.elastic.co/u/bukem)\
**Post date:** [August 13, 2018, 7:56pm UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/4 "2018-08-13T19:56:20Z")

</div>

> [@rjernst](#):
>
> I think you will need to use a script processor. None of the processors know how to deal with a nested field (which appears as an array of sub-maps). Eg, for doing lowercase with your example:
> 
> ```auto
> "processors" : [
> {
> "script" : {
> "source": "for (Map nested : ctx._source['nest']) { nested['field_lower'] = nested['field_raw'].toLowerCase(Locale.ROOT); }"
> }
> }
> ]
> 
> ```

That solution raises up the following error:

```auto
   "processors" : [
     {
       "script" : {
         "source": "for (Map nested : ctx._source['nest']) { nested['field_lower'] = nested['field_raw'].toLowerCase(Locale.ROOT); }"
       }
     }
   ]

{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "compile error",
        "script_stack" : [
          "... Map nested : ctx._source[nest]) { nested[field_low ...",
          " ^---- HERE"
        ],
        "script" : "for (Map nested : ctx._source[nest]) { nested[field_lower] = nested[field_raw].toLowerCase(Locale.ROOT); }",
        "lang" : "painless",
        "header" : {
          "processor_type" : "script",
          "property_name" : "source"
        }
      }
    ],
    "type" : "script_exception",
    "reason" : "compile error",
    "script_stack" : [
      "... Map nested : ctx._source[nest]) { nested[field_low ...",
      " ^---- HERE"
    ],
    "script" : "for (Map nested : ctx._source[nest]) { nested[field_lower] = nested[field_raw].toLowerCase(Locale.ROOT); }",
    "lang" : "painless",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Variable [nest] is not defined."
    },
    "header" : {
      "processor_type" : "script",
      "property_name" : "source"
    }
  },
  "status" : 500
}

```

I've tried to rewrite it to:

```auto
  "processors" : [
    {
      "script" : {
        "source": "for (Map nested : ctx._source.nest) { nested.field_lower = nested.field_raw.toLowerCase(Locale.ROOT); }"
      }
    }
  ]

```

and pipeline creation was done without errors, but indexing the document failed:

```auto
curl -X POST "localhost:9200/nest_test/1?pretty&refresh&pipeline=nested_lowercase" -H 'Content-Type: application/json' -d'
{"nest": [{"field_raw": "Test Me"}]}
'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "java.lang.IllegalArgumentException: ScriptException[runtime error]; nested: NullPointerException;",
        "header" : {
          "processor_type" : "script"
        }
      }
    ],
    "type" : "exception",
    "reason" : "java.lang.IllegalArgumentException: ScriptException[runtime error]; nested: NullPointerException;",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "ScriptException[runtime error]; nested: NullPointerException;",
      "caused_by" : {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : [
          "for (Map nested : ctx._source.nest) { ",
          " ^---- HERE"
        ],
        "script" : "for (Map nested : ctx._source.nest) { nested.field_lower = nested.field_raw.toLowerCase(Locale.ROOT); }",
        "lang" : "painless",
        "caused_by" : {
          "type" : "null_pointer_exception",
          "reason" : null
        }
      }
    },
    "header" : {
      "processor_type" : "script"
    }
  },
  "status" : 500
}

```

And other question is, if we'll succeed with lowercase, how can i replace JSON processor with scripting?

---

<div class="post-metadata">

**Author:** ![bukem](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@bukem](https://discuss.elastic.co/u/bukem)\
**Post date:** [August 13, 2018, 8:10pm UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/5 "2018-08-13T20:10:56Z")

</div>

> [@talevy](#):
>
> Another way to achieve this is with the help of the `foreach` processor.  
> ...

Nope, that doesn't work for me too:

```auto
curl -X PUT "localhost:9200/_ingest/pipeline/nested_lowercase?pretty" -H 'Content-Type: application/json' -d'
{
  "description" : "failure reproducing",
  "processors" : [
    {
      "foreach": {
        "field": "nest",
        "processor": {
          "lowercase": {
            "field": "_ingest._value.field_raw",
            "target_field": "_ingest._value.field_lower"
          }
        }
      }
    }]
}
'

curl -X PUT "localhost:9200/nest_test/_doc/1?pretty&refresh&pipeline=nested_lowercase" -H 'Content-Type: application/json' -d'
{"nest": [{"field_raw": "CamelCase"}]}
'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "object mapping [nest] can't be changed from nested to non-nested"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "object mapping [nest] can't be changed from nested to non-nested"
  },
  "status" : 400
}

```

I've changed POST method to PUT to have the same c=operations as yours, plus I've recreated the index without setting `"dynamic":"strict"` to the nested field, but all that didn't help.

---

<div class="post-metadata">

**Author:** ![talevy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talevy/32/44896_2.png) [@talevy](https://discuss.elastic.co/u/talevy)\
**Post date:** [August 13, 2018, 8:47pm UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/6 "2018-08-13T20:47:39Z")

</div>

> [@bukem](#):
>
> { "error" : { "root\_cause" : [{ "type" : "illegal\_argument\_exception", "reason" : "object mapping [nest] can't be changed from nested to non-nested" } ], "type" : "illegal\_argument\_exception", "reason" : "object mapping [nest] can't be changed from nested to non-nested" }, "status" : 400 }

This error leads me to believe there is a difference between the mapping you are running against, and the one I ran against. Since this mapping exception is due to actual indexing, this means that the ingest pipeline was successful in executing and this is a mismatch between the object being indexed and the index mapping.

would you mind sharing your response to `GET nest_test`, specifically the mappings?

---

<div class="post-metadata">

**Author:** ![bukem](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@bukem](https://discuss.elastic.co/u/bukem)\
**Post date:** [August 13, 2018, 9:24pm UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/7 "2018-08-13T21:24:47Z")

</div>

> [@talevy](#):
>
> would you mind sharing your response to `GET nest_test` , specifically the mappings?

It worked! The problem was with the type, that i missed while sending data to index.  
Incorrect in my case:

```auto
curl -X POST "localhost:9200/nest_test/_doc/1?pretty&refresh&pipeline=nested_lowercase" -H 'Content-Type: application/json' -d'
{"nest": [{"field_raw": "CamelCase"}]}
'

```

Correct:

```auto
curl -X POST "localhost:9200/nest_test/resources/1?pretty&refresh&pipeline=nested_lowercase" -H 'Content-Type: application/json' -d'
{"nest": [{"field_raw": "CamelCase"}]}
'

```

@talevy thank you very much!

I'll check all my variations tomorrow to check if `foreach` processor covers them all:

- applying two different processors to the same field, for example uppercase and lowercase, to store the result in two new fields
- accessing nested objects of nested object

---

<div class="post-metadata">

**Author:** ![bukem](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@bukem](https://discuss.elastic.co/u/bukem)\
**Post date:** [August 14, 2018, 8:31am UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/8 "2018-08-14T08:31:19Z")

</div>

It's OK with accessing nested object of nested object, but as you mentioned before there is no possibility to use `foreach` to apply more than one processor, thus that solution is not suitable for me.

> [@talevy](#):
>
> Since `foreach` only accepts one processor at a time to operate on the array elements, you may find it more customizable to use Painless scripting for additional transformations

Now, the only hope is to somehow make script processor work with nested objects.

I'll keep example of pipeline accessing nested-nested object using `foreach` Just in case someone will find it suitable:

```auto
  "processors" : [
    {
      "foreach": {
        "field": "nest",
        "processor": {
          "foreach" : {
            "field": "_ingest._value.inner_nest",
            "processor" : {
              "lowercase": {
                "field": "_ingest._value.inner_field_raw",
                "target_field": "_ingest._value.inner_field_lower"
              }
            }
          }
        }
      }
    }
  ]

```

and the corresponding document:  
`{"nest": [{"field_raw" : "Test Me", "inner_nest" : [{"inner_field_raw": "Test Me Inner"}]}]}`

---

<div class="post-metadata">

**Author:** ![bukem](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@bukem](https://discuss.elastic.co/u/bukem)\
**Post date:** [August 14, 2018, 9:57am UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/9 "2018-08-14T09:57:26Z")

</div>

> [@rjernst](#):
>
> I think you will need to use a script processor....

Just in case someone will need that, the correct solution will be to refer to the object using `ctx.field_name`, but not `ctx._source['field_name']` or `ctx._source.nest`. Thus, the correct script will look like:

```auto
  "processors" : [
    {
      "script" : {
        "source": "for (item in ctx.nest) { item.field_lower = item.field_raw.toLowerCase(Locale.ROOT); }"
      }
    }
  ]

```

My problem is still there, as I need not only [lowercase](https://github.com/elastic/elasticsearch/blob/master/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/LowercaseProcessor.java), but also [JSON](https://github.com/elastic/elasticsearch/blob/master/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/JsonProcessor.java) processor, which seems to be a little more complicated.

Anyway,  
@rjernst , @talevy thank you very much for the ideas!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 11, 2018, 9:57am UTC](https://discuss.elastic.co/t/ingest-processor-failure-when-accessing-nested-field/144194/10 "2018-09-11T09:57:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.