# Ingest pipeline

**URL:** https://discuss.elastic.co/t/ingest-pipeline/319931
**Category:** Elasticsearch
**Tags:** ingest-pipeline
**Created:** [November 28, 2022, 10:04am UTC](https://discuss.elastic.co/t/ingest-pipeline/319931 "2022-11-28T10:04:26Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![ramiwashere](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@ramiwashere](https://discuss.elastic.co/u/ramiwashere)
#### Post date: [November 28, 2022, 10:04am UTC](https://discuss.elastic.co/t/ingest-pipeline/319931/1 "2022-11-28T10:04:26Z")

</div>

Hi there,

Im new to ELK, and trying to extract some custom log no covered by filebeat module into kibana. So I tried to create a ingest node pipeline between filebeat and ES.

So on filebeat.yml: 1) Add the pipeline name on the elasticoutputsection 2) add the logpath for the custom log in the filebeat input

I created the ingest node where I'm using grok processors to parse vsftpd.log and it work with dev tools (where I had to transform the query from grok debugger because it need to be json format).

I want to test with the simulate but the "message" section isn't in json format. I don't understand why it should be in json format as the message is a redirect from vsftpd.log

Completlely lost on how it should work, even after reading documentation. On filebeat side, I see a line that pipeline output is connected to elastic, but no logs are redirect to the default filebeat index.

Here is the simulate output if anyone can help would be appreciate !

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": ["%{SYSLOGTIMESTAMP:timestamp}%{SPACE}+%{YEAR:year}%{SPACE}+.pid%{SPACE}+%{GREEDYDATA:pid}.%{SPACE}+.%{USERNAME:username}.%{SPACE}+%{GREEDYDATA:log_message}%{SPACE}","::ffff:%{IP:sourceIP}","%{SPACE}\"%{PATH:filepath}\",%{SPACE}%{GREEDYDATA:Packet_Size_and_Speed}"]
        }
      }
    ]
  },
  "docs":[
    {
      "_source": {
        "message": "Thu Nov 17 17:31:54 2022 [pid 7512] [rami] OK DOWNLOAD: Client "::ffff:192.168.1.98", "/home/rami/test2", 5 bytes, 8.91Kbyte/sec"
      }
    }
  ]
}

```

Thanks you

---

<div class="post-metadata">

### Author: ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)
#### Post date: [November 28, 2022, 12:35pm UTC](https://discuss.elastic.co/t/ingest-pipeline/319931/2 "2022-11-28T12:35:26Z")

</div>

Hi,

As your log message contains quotees you have to escape them correctly. The following request runs the simulation without errors:

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": ["%{SYSLOGTIMESTAMP:timestamp}%{SPACE}+%{YEAR:year}%{SPACE}+\\[pid%{SPACE}+%{NUMBER:pid}\\]%{SPACE}+\\[%{USERNAME:username}\\]%{SPACE}+%{DATA:log_message}%{SPACE}+\"::ffff:%{IP:sourceIP}\",%{SPACE}+\"%{PATH:filepath}\",%{SPACE}%{GREEDYDATA:Packet_Size_and_Speed}"]
        }
      }
    ]
  },
  "docs":[
    {
      "_source": {
        "message": "Thu Nov 17 17:31:54 2022 [pid 7512] [rami] OK DOWNLOAD: Client \"::ffff:192.168.1.98\", \"/home/rami/test2\", 5 bytes, 8.91Kbyte/sec"
      }
    }
  ]
}

```

It returns:

```auto
{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "sourceIP" : "192.168.1.98",
          "filepath" : "/home/rami/test2",
          "year" : "2022",
          "Packet_Size_and_Speed" : "5 bytes, 8.91Kbyte/sec",
          "pid" : "7512",
          "log_message" : "OK DOWNLOAD: Client",
          "message" : """Thu Nov 17 17:31:54 2022 [pid 7512] [rami] OK DOWNLOAD: Client "::ffff:192.168.1.98", "/home/rami/test2", 5 bytes, 8.91Kbyte/sec""",
          "timestamp" : "Nov 17 17:31:54",
          "username" : "rami"
        },
        "_ingest" : {
          "timestamp" : "2022-11-28T12:33:59.692194616Z"
        }
      }
    }
  ]
}

```

I hope this helps...

Best regards  
Wolfram

---

<div class="post-metadata">

### Author: ![ramiwashere](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@ramiwashere](https://discuss.elastic.co/u/ramiwashere)
#### Post date: [November 28, 2022, 1:19pm UTC](https://discuss.elastic.co/t/ingest-pipeline/319931/3 "2022-11-28T13:19:23Z")

</div>

Wolfram,

Thank you for the reply and for your help. Indeed, I misunderstood something with escape. I will retry it on an other log set.

Regards,

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 26, 2022, 1:20pm UTC](https://discuss.elastic.co/t/ingest-pipeline/319931/4 "2022-12-26T13:20:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.