# Issue with escaping pipe "|" in Ingest Pipeline

**URL:** <https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [October 8, 2021, 8:08pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279 "2021-10-08T20:08:08Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Thanura\_Kannangara](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\
**Post date:** [October 8, 2021, 8:08pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279/1 "2021-10-08T20:08:08Z")

</div>

Hi Guys,

I have a log patern that look like this  
(windows event log)  
`Dummy request has been approved.|Node=blah-blah.example.net_iam|BatchSig=jhsdjahsdgsjahdg|Requester=john.test|Recipient=ex0000123456`

below grok pattern seems to do the trick correctly

`%{DATA:winlog.event_data.Message}\|%{GREEDYDATA:kvpairs}`

EG:

 ![Screenshot_472](https://us1.discourse-cdn.com/elastic/original/3X/7/4/740d2e94b5ba9ea7cfddead87e93c28eab5b5545.jpeg)

However, in ingest pipeline. It does not let me save the pattern like that

 ![Screenshot_473](https://us1.discourse-cdn.com/elastic/original/3X/1/0/1046f5eb5cc5c047c23a04e48584d43f25643ea9.jpeg)

If I change the rule to  
`%{DATA:winlog.event_data.Message}\\|%{GREEDYDATA:kvpairs}`

which means with `\\|`, the JSON error goes away. but it seems the grok pattern is wrong with two `\\`

 ![Screenshot_474](https://us1.discourse-cdn.com/elastic/original/3X/d/d/ddf6116ae379626c608c885b364b60bb06a04ed5.jpeg)

How can I get this resolved?

My current pipeline with wrong grok pattern is this

```auto
PUT _ingest/pipeline/winlogbeat-hitachi
{
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{DATA:winlog.event_data.Message}\\|%{GREEDYDATA:kvpairs}"
        ],
        "if": "ctx?.winlog?.channel ==~ /Example*/",
        "ignore_failure": true
      }
    },
    {
      "kv": {
        "field": "kvpairs",
        "field_split": "\\|",
        "value_split": "=",
        "target_field": "winlog.event_data",
        "ignore_missing": true,
        "ignore_failure": true
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Thanura\_Kannangara](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\
**Post date:** [October 8, 2021, 8:11pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279/2 "2021-10-08T20:11:52Z")

</div>

> [@Thanura\_Kannangara](#):
>
> Dummy request has been approved.|Node=blah-blah.example.net\_iam|BatchSig=jhsdjahsdgsjahdg|Requester=john.test|Recipient=ex0000123456

Requirement i have is this

`Dummy request has been approved.|Node=blah-blah.example.net_iam|BatchSig=jhsdjahsdgsjahdg|Requester=john.test|Recipient=ex0000123456`

From this above log message  
I need to put them in fields as below.

```auto
message:Dummy request has been approved.
winlog.event_data.Node=blah-blah.example.net_iam
winlog.event_data.BatchSig=jhsdjahsdgsjahdg
winlog.event_data.Requester=john.test
winlog.event_data.Recipient=ex000012345

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 8, 2021, 8:29pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279/3 "2021-10-08T20:29:40Z")

</div>

Have you tried without escaping it? I don't think there is any need to escape the `|` in an ingest pipeline or in logstash.

You also do not need `grok`, the `dissect` processor can do the same thing and use less CPU.

Try this:

```auto
{
    "dissect": {
        "field": "message",
        "pattern" : "%{winlog.event_data.Message}|%{kvpairs}",
        "if": "ctx?.winlog?.channel ==~ /Example*/",
        "ignore_failure": true
    }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 8, 2021, 8:37pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279/4 "2021-10-08T20:37:07Z")

</div>

Just a drive by thought, dissect + KV processor will probably be more computationally efficient not to mention easier.

This works probably faster computationally than grok.  
You can remove the uneccesary

```auto
POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
  "processors": [
    {
      "dissect": {
        "field": "raw_message",
        "pattern": "%{message}|%{key_values}"
      }
    },
    {
      "kv": {
        "field": "key_values",
        "field_split": "\\|",
        "value_split": "=",
        "target_field": "winlog.event_data"
      }
    }
  ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "raw_message" : "Dummy request has been approved.|Node=blah-blah.example.net_iam|BatchSig=jhsdjahsdgsjahdg|Requester=john.test|Recipient=ex0000123456"
      }
    }
  ]
}

```

Results

```auto
{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "winlog" : {
            "event_data" : {
              "BatchSig" : "jhsdjahsdgsjahdg",
              "Requester" : "john.test",
              "Recipient" : "ex0000123456",
              "Node" : "blah-blah.example.net_iam"
            }
          },
          "raw_message" : "Dummy request has been approved.|Node=blah-blah.example.net_iam|BatchSig=jhsdjahsdgsjahdg|Requester=john.test|Recipient=ex0000123456",
          "key_values" : "Node=blah-blah.example.net_iam|BatchSig=jhsdjahsdgsjahdg|Requester=john.test|Recipient=ex0000123456",
          "message" : "Dummy request has been approved."
        },
        "_ingest" : {
          "timestamp" : "2021-10-08T20:34:59.240685423Z"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Thanura\_Kannangara](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\
**Post date:** [October 8, 2021, 8:41pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279/5 "2021-10-08T20:41:53Z")

</div>

> [@leandrojmp](#):
>
> `"ctx?.winlog?.channel ==~ /Example*/"`

Thanks a lot @leandrojmp  
My condition is based on this field,

 ![Screenshot_476](https://us1.discourse-cdn.com/elastic/original/3X/6/4/6472c66f312de5ed0594703f51ac08b54ab6f8d7.jpeg)

Do I have it incorrectly configured by doing this?

`ctx?.winlog?.channel ==~ /example*/`  
??

---

<div class="post-metadata">

**Author:** ![Thanura\_Kannangara](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\
**Post date:** [October 8, 2021, 9:03pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279/6 "2021-10-08T21:03:36Z")

</div>

Confirmed This works.

```auto
PUT _ingest/pipeline/winlogbeat-example
{
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{message}|%{kvpairs}",
        "ignore_failure": true
      }
    },
    {
      "kv": {
        "field": "kvpairs",
        "field_split": "\\|",
        "value_split": "=",
        "target_field": "winlog.event_data",
        "ignore_missing": true,
        "ignore_failure": true
      }
    }
  ]
}

```

I have an issue with the `Condition`

In the actual documents, I have field values like this

```auto
winlog.channel:Example-Example ID Systems-Example ID Suite/Operational
winlog.channel:Example-Example ID Systems-Example ID Suite/Admin

```

And I need to apply above processor only for these events.  
How can I do this?  
Right now, what I have is this. Which is clearly not working  
` "if": "ctx?.winlog?.channel ==~ /Example*/",`

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 8, 2021, 9:09pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279/7 "2021-10-08T21:09:19Z")

</div>

I think that the `==~` is wrong, the operator is just `=~`.

But you also can try something like this:

```auto
"if": "ctx.winlog?.channel?.contains('Example')

```

---

<div class="post-metadata">

**Author:** ![Thanura\_Kannangara](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\
**Post date:** [October 8, 2021, 9:20pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279/8 "2021-10-08T21:20:37Z")

</div>

Thanks a lot guys @leandrojmp @stephenb .  
I have a working pipeline now. You rock 👊 👊 🤟 🤟

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 5, 2021, 9:20pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279/9 "2021-11-05T21:20:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.