# Foreach Ingest processor + conditional append processor

**URL:** <https://discuss.elastic.co/t/foreach-ingest-processor-conditional-append-processor/216884>\
**Category:** Elasticsearch\
**Tags:** painless\
**Created:** [January 28, 2020, 4:53pm UTC](https://discuss.elastic.co/t/foreach-ingest-processor-conditional-append-processor/216884 "2020-01-28T16:53:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kredit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kredit/32/33089_2.png) [@Kredit](https://discuss.elastic.co/u/Kredit)\
**Post date:** [January 28, 2020, 4:53pm UTC](https://discuss.elastic.co/t/foreach-ingest-processor-conditional-append-processor/216884/1 "2020-01-28T16:53:05Z")

</div>

I have documents with fields that looks like this (from [Jaeger](https://www.jaegertracing.io/)):

```json
{
	"something": "some value",
	...
	"logs": [
		{
			"fields": [
				{
					"key": "error",
					"value": "This is an error message."
				},
				{
					"key": "info",
					"value": "Arbitrary info"
				}
			]
		}
	]
}

```

I am trying to add an ingest processor to create a new field `log_values` containing just the value of each field. So far I have this working perfectly with two foreach processors:

```json
{
	"foreach": {
		"field": "logs",
		"processor": {
			"foreach": {
				"field": "_ingest._value.fields",
				"processor": {
					"append": {
						"field": "log_values",
						"value": ["{{_ingest._value.value}}"]
					}
				}
			}
		}
	}
}

```

This results in a new field being created in the indexed document:

```json
{
	"something": "some value",
	...
	"logs": [...],
	"log_values": ["This is an error message.", "Arbitrary info"]
}

```

However, I only want to append logs where the `key` field is `error`. So I tried the obvious:

```json
"append": {
	"field": "log_values",
	"if": "_ingest._value.key == 'error'",
	"value": ["{{_ingest._value.value}}"]
}

```

Elasticsearch doesn't like this though, as I now get errors such as `Variable [_ingest] is not defined.` I've made a few more attempts using `ctx`, but nothing has worked and I'm really not sure where to go next.

Is what I'm trying to do possible? Thank you for your time.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 29, 2020, 3:48pm UTC](https://discuss.elastic.co/t/foreach-ingest-processor-conditional-append-processor/216884/2 "2020-01-29T15:48:42Z")

</div>

Try a `script` processor...

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": """
ctx.error_values = ctx.logs.stream().map(log -> 
  log.fields.stream()
    .filter(field -> field.key == "error")
    .map(field -> field.value)
    .collect(Collectors.toList())
  )
  .flatMap(l -> l.stream())
  .collect(Collectors.toList())"""
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "something": "some value",
        "logs": [
          {
            "fields": [
              {
                "key": "error",
                "value": "This is an error message."
              },
              {
                "key": "info",
                "value": "Arbitrary info"
              },
              {
                "key": "error",
                "value": "This is a second error message."
              }
            ]
          }
        ]
      }
    },
    {
      "_source": {
        "something": "some value",
        "logs": [
          {
            "fields": [
              {
                "key": "error",
                "value": "This is a third error message."
              },
              {
                "key": "info",
                "value": "Arbitrary info"
              }
            ]
          }
        ]
      }
    }
    
  ]
}

```

note, that the triple double ticks syntax is from kibana and might ne to be replaced when used with curl.

---

<div class="post-metadata">

**Author:** ![Kredit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kredit/32/33089_2.png) [@Kredit](https://discuss.elastic.co/u/Kredit)\
**Post date:** [January 29, 2020, 3:52pm UTC](https://discuss.elastic.co/t/foreach-ingest-processor-conditional-append-processor/216884/3 "2020-01-29T15:52:00Z")

</div>

Wow, thank you! I was trying to get it done with the built-in filters, but this works perfectly and is easier to maintain as well. I really need to learn how to use scripts in ES.

Thank you again!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 26, 2020, 3:52pm UTC](https://discuss.elastic.co/t/foreach-ingest-processor-conditional-append-processor/216884/4 "2020-02-26T15:52:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.