# Conditionals in pipelines

**URL:** <https://discuss.elastic.co/t/conditionals-in-pipelines/294970>\
**Category:** Kibana\
**Tags:** ingest-pipeline\
**Created:** [January 20, 2022, 3:57pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970 "2022-01-20T15:57:15Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tuckson](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Tuckson](https://discuss.elastic.co/u/Tuckson)\
**Post date:** [January 20, 2022, 3:57pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/1 "2022-01-20T15:57:15Z")

</div>

Hi,

We are in the process of recreating our logstash pipelines in kibana/elk to use Elasticsearch pipelines instead of logstash. However the lack of documentation on how to do this in Kibana is overwhelming.

See the accompanied picture. What is the syntax for providing a simple 'this field contains that value' conditional here? I searched for this question, but prior questions on this (like [Condition format for pipelines in Kibana](https://discuss.elastic.co/t/condition-format-for-pipelines-in-kibana/265985) ) got no answers.

Since at least somebody designed this, there should be an answer somewhere though 😉

Does somebody know? Whatever I come up with I keep getting compile errors (yes, also tried without parenthesis).

 ![msedge_jDEAuyeg00](https://us1.discourse-cdn.com/elastic/original/3X/f/f/ffaf21b1ab77bfe60dc8ec17de14e718b13a7683.png)

However, when I put a pipeline from somwhere in the docs this seems rather similar. What am I doing wrong?

![image](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bb00b1113cb27a600562ebe3648fdb4a7fb82f34.png)

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [January 21, 2022, 9:59pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/2 "2022-01-21T21:59:27Z")

</div>

> **[Ingest pipelines | Elasticsearch Guide \[7.16\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#conditionally-run-processor)**

This document help you?

---

<div class="post-metadata">

**Author:** ![Tuckson](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Tuckson](https://discuss.elastic.co/u/Tuckson)\
**Post date:** [January 21, 2022, 10:15pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/3 "2022-01-21T22:15:26Z")

</div>

No, sorry.  
I've seen this page and even importted one of the pipelines to see how it is shown jn kibana.  
It just doesnt work with my own pipeline.

Regards

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [January 21, 2022, 10:30pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/4 "2022-01-21T22:30:12Z")

</div>

> **[Shared API for package org.elasticsearch.index.fielddata | Painless Scripting...](https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-api-reference-shared-org-elasticsearch-index-fielddata.html#painless-api-reference-shared-ScriptDocValues-Strings)**

ok, there is `contains` function for string value. Have you tried this?

And what was the result of your script? Error message or undesired result?

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [January 22, 2022, 12:09pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/5 "2022-01-22T12:09:30Z")

</div>

> If the [`script.painless.regex.enabled`](https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker.html#script-painless-regex-enabled) cluster setting is enabled, you can use regular expressions in your `if` condition scripts. For supported syntax, see [Painless regular expressions](https://www.elastic.co/guide/en/elasticsearch/painless/7.16/painless-regexes.html).

Check that?

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [January 22, 2022, 12:13pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/6 "2022-01-22T12:13:32Z")

</div>

It also could be this if you are trying to match `http` only.

`"if": "ctx.url?.scheme =~ /^http[^s]/"`  
should be  
`"if": "ctx.url?.scheme == /^http[^s]/"`

You can put these in Dev Tools to test the results to see if they work or not.

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "if": "ctx.url =~ /^http[^s]/",
          "field": "result",
          "value": true
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "url": "http://www.google.com"
      }
    },
    {
      "_source": {
        "url": "https://www.google.com"
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Tuckson](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Tuckson](https://discuss.elastic.co/u/Tuckson)\
**Post date:** [January 24, 2022, 1:01pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/7 "2022-01-24T13:01:22Z")

</div>

Thnx for the replies.  
What I am trying to do is simply match the value of a field. I have a field 'event-type'. This field can contain the string 'API\_CALL' . I want the dissect from my OP to only be tried when event-type == 'API\_CALL'

The funny thing is, I have a few processors in the pipeline already. When I save, it's OK and it processes the document I feed it. When I go to devtools en do GET \_ingest/pipeline/pipelinename I get the pipeline in return. But when I then do PUT \_ingest/pipeline/my-pipeline with te exact json I just got returned, I get a parse exception. Funny.

I already found out I somehow seem to need to use that (horrible) painless stuff. Appearantly I need to use ctx as source and then elaborate on that. However just: ctx.event-type == API\_CALL still gives a compile error.

I tried aaron\_nimocks suggestion:

```auto
POST _ingest/pipeline/AAB_AGL/_simulate
{
  "AAB_AGL" : {
    "processors" : [
      {
        "dissect" : {
          "field" : "message",
          "pattern" : "ts: %{ts} | logLevel: %{log-level} | appId: %{app-id} | thread: %{thread-id} | SID: %{session-id} | TN: %{transaction-id} | clientIp: %{client-ip} | userId: %{user-id} | apiType: %{api-type} | api: %{api-url} | platform: %{platform} | eventType: %{event-type} | %{additional-data}"
        }
      },
      {
        "trim" : {
          "field" : "app-id",
          "ignore_failure" : true
        }
      },
      {
        "trim" : {
          "field" : "client-ip",
          "ignore_failure" : true
        }
      },
      {
        "trim" : {
          "field" : "api-type",
          "ignore_failure" : true
        }
      },
      {
        "trim" : {
          "field" : "api-url",
          "ignore_failure" : true
        }
      },
      {
        "dissect" : {
          "field" : "additional-data",
          "pattern" : "message: %{ms-url}|%{ms-result-code}|%{ms-result}|%{execution-time}",
          "if": "ctx.event-type == API_CALL",
          "value": true
        }
      }
    ]
  }
}

```

But this results in:

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "[docs] required property is missing",
        "property_name" : "docs"
      }
    ],
    "type" : "parse_exception",
    "reason" : "[docs] required property is missing",
    "property_name" : "docs"
  },
  "status" : 400
}

```

Feel free to try.  
This is an anonimised document I test with.

```auto
{
  "_source": {
  "@timestamp": "2022-01-20T12:56:45.262Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.16.2"
  },
  "fields": {
    "environment": "production"
  },
  "agent": {
    "hostname": "server1.prdl.itv.local",
    "ephemeral_id": "866496c4-e379-421e-930c-1ade47f5105c",
    "id": "a86414cc-90f1-4b71-9723-d115033864d7",
    "name": "server1.prdl.itv.local",
    "type": "filebeat",
    "version": "7.16.2"
  },
  "ecs": {
    "version": "1.12.0"
  },
  "message": "ts: 2022-01-20 13:56:44.299 | logLevel: INFO | appId: AGL | thread: 111309 | SID: 1e7ad1c0-e99a-7af6-edd4-a3384bd19247 | TN: a39ffed3-7120-6313-ab67-045ee0ef6f20 | clientIp: 127.0.0.1 | userId: 0000000 | apiType: NANO | api: POST /100/1.2.0/A/nld/stb/kpn/API-1/MS-1 | platform: stb | eventType: API_CALL | message: http://newservername:8080/new-api-name/b2b/tokens?channel=stb&lang=nld|20X|ACN_200|5",
  "log": {
    "file": {
      "path": "/product/AGL/agl-core/logs/agl.log"
    },
    "offset": 2884772107
  },
  "tags": [
    "avs6",
    "api-log",
    "apigateway",
    "asd"
  ],
  "input": {
    "type": "log"
  }
}
}

```

---

<div class="post-metadata">

**Author:** ![Tuckson](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Tuckson](https://discuss.elastic.co/u/Tuckson)\
**Post date:** [January 24, 2022, 1:05pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/8 "2022-01-24T13:05:13Z")

</div>

Tried this one. Again a compile error.

```auto
ctx.event-type.contains('API_CALL')

```

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [January 24, 2022, 1:14pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/9 "2022-01-24T13:14:06Z")

</div>

> [@Tuckson](#):
>
> ```auto
> POST _ingest/pipeline/AAB_AGL/_simulate
> 
> ```

The error you are seeing is because you are trying to `_simulate` the processor which requires `docs` to sample from.

Change to `POST _ingest/pipeline/AAB_AGL/` if you are trying to save the pipeline vs simulating it.

---

<div class="post-metadata">

**Author:** ![Tuckson](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Tuckson](https://discuss.elastic.co/u/Tuckson)\
**Post date:** [January 24, 2022, 3:37pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/10 "2022-01-24T15:37:05Z")

</div>

That gives a 405

```auto
{
  "error" : "Incorrect HTTP method for uri [/_ingest/pipeline/AAB_AGL?pretty=true] and method [POST], allowed: [PUT, GET, DELETE]",
  "status" : 405
}

```

And after changing it to PUT,  
I get:

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "[processors] required property is missing",
        "property_name" : "processors"
      }
    ],
    "type" : "parse_exception",
    "reason" : "[processors] required property is missing",
    "property_name" : "processors"
  },
  "status" : 400
}

```

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [January 24, 2022, 4:16pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/11 "2022-01-24T16:16:36Z")

</div>

Sorry. Here's a full example if this helps.

```auto
PUT _ingest/pipeline/my-pipeline
{
  "processors": [
    {
      "set": {
        "description": "If 'url.scheme' is 'http', set 'url.insecure' to true",
        "if": "ctx.url =~ /^http[^s]/",
        "field": "result",
        "value": true
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Tuckson](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Tuckson](https://discuss.elastic.co/u/Tuckson)\
**Post date:** [January 24, 2022, 4:17pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/12 "2022-01-24T16:17:49Z")

</div>

OK,

Finally got this syntax right.  
There is a f\*\*\*ing lot of documentation on ALL kinds of complex stuff with elastic, but I finally needed some external blog of 2 years ago to find my solution. Please @elastic , make parts of your documentation MORE kibana oriënted AND more beginner friendly.

OK, proper syntax for the conditional field in my case is:

```auto
ctx['event-type'] == 'API_CALL'

```

This did the trick.

Thnx for being with me.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2022, 4:18pm UTC](https://discuss.elastic.co/t/conditionals-in-pipelines/294970/13 "2022-02-21T16:18:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.