# Injest Pipeline for conditional statement used in Scripts

**URL:** <https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178>\
**Category:** Kibana\
**Tags:** painless, kql-kibana-query-language, ingest-pipeline\
**Created:** [February 24, 2022, 1:59pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178 "2022-02-24T13:59:00Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 24, 2022, 1:59pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/1 "2022-02-24T13:59:00Z")

</div>

Hii,

My goals -

1. Find the existing field if present `(responseBody.results[i].statusCode)` and copy it's value into a new field `(embedded_error_code )` at **runtime**
2. My original field is a list of array. Below is the structure of my original field - `(responseBody.results[i].statusCode)`

```auto
 "responseBody": {
      "results": [
        {
          "hasErrors": false,
          "statusCode": 201,
          "uniqueId": "***",
          "messageId": null,
          "customerId": "***",
          "developerMessage": "SUCCESS",
          "userMessage": "SUCCESS",
          "errors": []
        }
      ]
    },

```

My working Script when used with `_update_by_query`

```auto
POST _scripts/test-embedded-script
{
  "script": {
    "lang" : "painless",
     "source": """
        if(ctx._source.responseBody != null )
        {
            for (int i = 0; i < ctx._source.responseBody.results.length; ++i) 
            {
              ctx._source.embedded_error_code = ctx._source.responseBody.results[i].statusCode;
            } 
        }"""
  }
}

```

My defined pipeline -

```auto
PUT _ingest/pipeline/apiproxy-embedded
{
  "processors": [
    {
      "script": {
        "id": "test-embedded-script"
      }
    }
  ]
}

```

I want to run the above script runtime, that's why I'm using Injest Pipeline  
`test-embedded-script` isn't working in `apiproxy-embedded` pipeline.  
Please help me to write this script as one of the processors of my pipeline. 🙏

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 24, 2022, 2:36pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/2 "2022-02-24T14:36:37Z")

</div>

Ingest pipline works literally while ingesting. If you want to run the script at query runtime, use runtime mappings in the query. "[Define runtime fields in a search request](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-search-request.html)" may help you.

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 24, 2022, 5:14pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/3 "2022-02-24T17:14:32Z")

</div>

Thank you @Tomo_M, I'll try this and will update.  
Just a quick question about invest pipelines, why I shouldn't use invest over runtime fields?

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 24, 2022, 5:57pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/5 "2022-02-24T17:57:47Z")

</div>

@Tomo_M I tried this and as per my understanding this only works if my existing field is mapped or has to be mapped? As per my need I don't want to map my existing fields `(responseBody.results[i].statusCode)` but new field `embedded_error_code`.

Because Injest pipeline worked in my case that gives me new single field without mapping my existing field which might be a json array, I inclined towards it. Any other solution please help 🙏

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 25, 2022, 9:10pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/6 "2022-02-25T21:10:16Z")

</div>

> [@shivani\_singh](#):
>
> at **runtime**

What did you mean by "at runtime"?

> In most cases, retrieve field values through doc\_values whenever possible. Accessing doc\_values with a runtime field is faster than retrieving values from \_source because of how data is loaded from Lucene.  
> However, there are cases where retrieving fields from \_source is necessary. For example, text fields do not have doc\_values available by default, so you have to retrieve values from \_source. In other instances, you might choose to disable doc\_values on a specific field.  
> [Map a runtime field | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-mapping-fields.html)

Accessing `_source` in runtime field is also possible.

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 27, 2022, 4:03pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/7 "2022-02-27T16:03:26Z")

</div>

Hi @Tomo_M,  
**at runtime** - I mean doing this all stuff while data takes defined mapping or default pipeline configuration from **Index Template**.

Now,  
**Why runtime field didn't work for me** - this field `(responseBody.results[i].statusCode)` has different data type in different document for eg. my logs are coming from different apiProxies so that's why I get different responseBody and sometimes it's a keyword and sometimes it's an object.

Injest pipeline worked for my other cases where I didn't have list in my responseBody but not here - `(responseBody.results[i].statusCode)` because **result** is a list.

As you suggested before, I defined this mapping -

```auto
"mappings": {
      "runtime": {
        "runtime_field_24": {
          "type": "keyword",
          "script": {
            "source": "emit(doc['responseBody']['results']['statusCode'].value)"
          }
        }
      }
}

```

> But I got this error -  
> ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/f/4f4dcdb93538ec300a3ed9acf4403cd3054697df.png)

**Let me know if did any mistake defining this mapping or can we do this via Injest pipeline?**

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 27, 2022, 4:13pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/8 "2022-02-27T16:13:56Z")

</div>

> [@shivani\_singh](#):
>
> **at runtime** - I mean doing this all stuff while data takes defined mapping or default pipeline configuration from **Index Template**.

It is not an elastic meaning of runtime. I suppose it is better to avoid using another meaning of runtime to avoid confusion.

> [@shivani\_singh](#):
>
> But I got this error -

Please share the error by text, it's difficult to read screenshot. It is not seachable eighter.

That said, the error caused by the '`responseBody`' field type is not consistent with mappings. The error will occure even without the runtime mapping.

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 27, 2022, 5:23pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/9 "2022-02-27T17:23:02Z")

</div>

> [@Tomo\_M](#):
>
> ror will occure even without the runtime mapping.

Error above in screenshot-  
**Object mapping for [responseBody] tried to parse field [responseBodt] as object, but foound a concrete value**

> That said, the error caused by the ' `responseBody` ' field type is not consistent with mappings. The error will occure even without the runtime mapping.

To answer above statement -

That's why I'm not indexing this field (`responseBody`) in my index pattern so that I don't get conflicted field issue (occurs when same field has different data type in the single index pattern)

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 27, 2022, 10:09pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/10 "2022-02-27T22:09:30Z")

</div>

Your pipeline just add new fields to the \_source.

Use remove processor to remove the other fields, or just refresh `ctx._source` in your script.

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 28, 2022, 4:36am UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/11 "2022-02-28T04:36:42Z")

</div>

Hi @Tomo_M ,  
Are you suggesting me to write a **1. script** like this and then define a **2. processo** r in my pipeline -

```auto
POST _scripts/test-embedded-script
{
  "script": {
    "lang" : "painless",
     "source": """
        if(ctx._source.responseBody != null )
        {
            for (int i = 0; i < ctx._source.responseBody.results.length; ++i) 
            {
              ctx._source.embedded_error_code = ctx._source.responseBody.results[i].statusCode;
            } 
        }"""
  }
}

```

```auto
PUT _ingest/pipeline/apiproxy-embedded
{
  "processors": [
    {
      "script": {
        "id": "test-embedded-script"
      }
    }
  ]
}

```

I didn't understand -

> Your pipeline just add new fields to the \_source.

It would be really helpful if you could just point out to exact line I have change or modify in my above script/ pipeline.

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 28, 2022, 8:41am UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/12 "2022-02-28T08:41:34Z")

</div>

> [@shivani\_singh](#):
>
> That's why I'm not indexing this field ( `responseBody` ) in my index pattern so that I don't get conflicted field issue (occurs when same field has different data type in the single index pattern)

Maybe you do not turn off [dynamic mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html) and the field type mapping for `responseBody` was created at the first document indexed. Then error occured when documents containing other type `responseBody` were to be indexed.

> [@Tomo\_M](#):
>
> Your pipeline just add new fields to the \_source.

Than meant your processor does not remove the original fields.

Try:

```auto
POST _scripts/test-embedded-script
{
  "script": {
    "lang" : "painless",
     "source": """
        if(ctx._source.responseBody != null )
        {
            for (int i = 0; i < ctx._source.responseBody.results.length; ++i) 
            {
              ctx._source.embedded_error_code = ctx._source.responseBody.results[i].statusCode;
            } 
        }"""
  }
}

PUT _ingest/pipeline/apiproxy-embedded
{
  "processors": [
    {
      "script": {
        "id": "test-embedded-script"
      }
    },
    {
      "remove": {
        "field": ["responseBody "]
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 28, 2022, 12:56pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/13 "2022-02-28T12:56:30Z")

</div>

Hi @Tomo_M ,  
Even after trying above script and pipeline processor as you mentioned I'm getting this error while I'm reindexing my data for testing-

```auto
  "failures" : [
    {
      "index" : "test-apiproxy-2022.02.02",
      "type" : "_doc",
      "id" : "EaXcvH4B1BqwuQQQ8qv3",
      "cause" : {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : [
          "if(ctx._source.responseBody != null )\r\n {\r\n ",
          " ^---- HERE"
        ],
        "script" : "embeddedcode-script",
        "lang" : "painless",
        "position" : {
          "offset" : 24,
          "start" : 10,
          "end" : 72
        },
        "caused_by" : {
          "type" : "null_pointer_exception",
          "reason" : "cannot access method/field [responseBody] from a null def reference"
        }
      },
      "status" : 400
    },
]

```

I don't understand the null pointer error even after checking this condition in my script

```auto
if(ctx._source.responseBody != null )

```

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 28, 2022, 1:03pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/14 "2022-02-28T13:03:14Z")

</div>

Check if `ctx._source.containsKey("responseBody")`.

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 28, 2022, 1:16pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/15 "2022-02-28T13:16:30Z")

</div>

Getting this error -

```auto
    {
      "index" : "test-apiproxy-2022.02.02",
      "type" : "_doc",
      "id" : "EaXcvH4B1BqwuQQQ8qv3",
      "cause" : {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : [
          "if(ctx._source.containsKey(\"responseBody\"))\r\n {\r\n ",
          " ^---- HERE"
        ],
        "script" : "embeddedcode-script",
        "lang" : "painless",
        "position" : {
          "offset" : 24,
          "start" : 10,
          "end" : 78
        },
        "caused_by" : {
          "type" : "null_pointer_exception",
          "reason" : "cannot access method/field [containsKey] from a null def reference"
        }
      },
      "status" : 400
    },

```

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 28, 2022, 2:02pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/16 "2022-02-28T14:02:33Z")

</div>

Oh sorry, You have to use `ctx['my-field']` or `ctx.<my-field>` syntax in ingest pipeline script processor.

> **[Script processor | Elasticsearch Guide \[8.0\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html#script-processor-access-source-fields)**

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 28, 2022, 2:34pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/17 "2022-02-28T14:34:17Z")

</div>

Tested with this script although this time I didn't get any error but I didn't get result in `embedded_error_code` either. Is there any mistakes in below script ?  
Note : I defined this script directly in Injest pipeline processor list

```auto
    {
      "script": {
        "description": "testing responseBody script again",
        "lang": "painless",
        "source": """
        if( ctx.containsKey(['responseBody']))
        {
            for (int i = 0; i < ctx['responseBody']['results'].value.size(); ++i) 
            {
              ctx['embedded_error_code'] = ctx['responseBody']['results'][i]['statusCode'];
            } 
        }"""
      }
    }

```

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 28, 2022, 3:13pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/18 "2022-02-28T15:13:52Z")

</div>

> [@shivani\_singh](#):
>
> `if( ctx.containsKey(['responseBody']))`

should be `if( ctx.containsKey('responseBody'))`  
But I'm not sure if there is containsKey method in ctx.

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 28, 2022, 6:08pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/19 "2022-02-28T18:08:21Z")

</div>

Using this script, not getting null pointer exception while reindexing but still not getting values `['responseBody']['results'][i]['statusCode']`of in `[embedded_error_code]`

```auto
POST _scripts/embeddedcode_script
{
  "script": {
    "description": "testing responseBody script again",
    "lang": "painless",
    "source": """
        if( ctx['responseBody'] != null && ctx.containsKey('results'))
        {
            for (int i = 0; i < ctx['responseBody']['results'].value.size(); ++i) 
            {
              ctx['embedded_error_code'] = ctx['responseBody']['results'][i]['statusCode'];
            } 
        }"""
  }
}

```

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [February 28, 2022, 6:48pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/20 "2022-02-28T18:48:34Z")

</div>

I'm understanding a mistake here why I'm not getting values in `embedded_error_code`

When I tested a document in pipeline for testing pipeline, `Script` got executed but I guess this line is not working here in `Script` as it worked for` _update_query` -

```auto
 ctx['embedded_error_code'] = ctx['responseBody']['results'][i]['statusCode'];

```

**Any idea how I can assign existing field's value into another field inside Injested pipeline Script?**

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 28, 2022, 11:09pm UTC](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178/21 "2022-02-28T23:09:16Z")

</div>

> [@shivani\_singh](#):
>
> `ctx['embedded_error_code']` = ...

is the way to set a value on that field. But your script overwrite the value in each iteration.

I'm not sure what you want to do, it is possible to create an array and add values in each iteration, then finally set the array to `ctx['embedded_error_code']`.

[Next page](https://discuss.elastic.co/t/injest-pipeline-for-conditional-statement-used-in-scripts/298178.md?page=2)