# Elastic Ingest with multiple grok processors

**URL:** <https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451>\
**Category:** Elasticsearch\
**Created:** [January 3, 2017, 3:27pm UTC](https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451 "2017-01-03T15:27:42Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![JamesFx](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JamesFx](https://discuss.elastic.co/u/JamesFx)\
**Post date:** [January 3, 2017, 3:27pm UTC](https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451/1 "2017-01-03T15:27:42Z")

</div>

Happy New Year everyone!

I am trying to configure an ingest pipeline and I want to apply grok to two distinct fields. I couldn't find a way to do this. Does anyone have a suggestion on how to accomplish this.

Thank you!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 3, 2017, 3:40pm UTC](https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451/2 "2017-01-03T15:40:51Z")

</div>

You can add 2 grok processors (one for each field) in the pipeline.

---

<div class="post-metadata">

**Author:** ![JamesFx](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JamesFx](https://discuss.elastic.co/u/JamesFx)\
**Post date:** [January 3, 2017, 4:06pm UTC](https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451/3 "2017-01-03T16:06:08Z")

</div>

That was my first try.

I am using version 5.1 and it only takes into account the last grok.

```
"processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{STATUS:status};\\s+%{WORD:service};\\s+%{MSG:message}\\|\\s+%{WORD:key01}=%{NUMBER:value01}.*",
          "%{STATUS:status};\\s+%{WORD:service};\\s+%{GREEDYDATA:message}"
        ],
        "pattern_definitions": {
          "STATUS": "\\d+",
          "MSG": ".+?"
        }
      },
      "grok": {
          "field": "source",
          "patterns": [".+?%{TIMESTAMP:timestamp}.+"],
          "pattern_definitions" : {
            "TIMESTAMP" : "[0-9]+"}
        }
      ,
      "date" : {
        "field" : "timestamp",
        "formats" : ["UNIX_MS"]
      },
      "remove": {
        "field": "timestamp"
      }
    }
  ]
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 3, 2017, 4:24pm UTC](https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451/4 "2017-01-03T16:24:40Z")

</div>

Can you check with `verbose` parameter if only one grok is actually applied?

See [https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html#ingest-verbose-param](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html#ingest-verbose-param)

---

<div class="post-metadata">

**Author:** ![JamesFx](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JamesFx](https://discuss.elastic.co/u/JamesFx)\
**Post date:** [January 3, 2017, 6:10pm UTC](https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451/5 "2017-01-03T18:10:39Z")

</div>

My processors:

```
"processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{STATUS:status};\\s+%{WORD:service};\\s+%{MSG:message}\\|\\s+%{WORD:key01}=%{NUMBER:value01}.*",
          "%{STATUS:status};\\s+%{WORD:service};\\s+%{GREEDYDATA:message}"
        ],
        "pattern_definitions": {
          "STATUS": "\\d+",
          "MSG": ".+?"
        }
      },
      "grok": {
        "field": "source",
        "patterns": [
          ".+?%{TIMESTAMP:timestamp}.+"
        ],
        "pattern_definitions": {
          "TIMESTAMP": "[0-9]+"
        }
      },
      "date": {
        "field": "timestamp",
        "formats": [
          "UNIX_MS"
        ]
      }
    }
  ]

```

My Doc:

```
  "docs": [
    {
      "_index": "filebeat-test-2017.01.03",
      "_type": "log",
      "_id": "AVlkvVA78rJBRllM9MQ-",
      "_source": {
        "source": "/opt/shared-nagios/nagios_file_1483453815014.txt",
        "message": "0; SERVICE_A; online | time=1s",
        "@timestamp": "2017-01-03T14:30:15.311Z"
        }
      ,
      "fields": {
        "@timestamp": [
          1483453815311
        ]
      },
      "sort": [
        1483453815311
      ]
    }
  ]

```

My result:

```
{
  "docs": [
    {
      "processor_results": [
        {
          "doc": {
            "_id": "AVlkvVA78rJBRllM9MQ-",
            "_index": "filebeat-test-2017.01.03",
            "_type": "log",
            "_source": {
              "@timestamp": "2017-01-03T14:30:15.311Z",
              "source": "/opt/shared-nagios/nagios_file_1483453815014.txt",
              "message": "0; SERVICE_A; online | time=1s",
              "timestamp": "1483453815014"
            },
            "_ingest": {
              "timestamp": "2017-01-03T18:10:23.455+0000"
            }
          }
        },
        {
          "doc": {
            "_id": "AVlkvVA78rJBRllM9MQ-",
            "_index": "filebeat-test-2017.01.03",
            "_type": "log",
            "_source": {
              "@timestamp": "2017-01-03T14:30:15.014Z",
              "source": "/opt/shared-nagios/nagios_file_1483453815014.txt",
              "message": "0; SERVICE_A; online | time=1s",
              "timestamp": "1483453815014"
            },
            "_ingest": {
              "timestamp": "2017-01-03T18:10:23.455+0000"
            }
          }
        }
      ]
    }
  ]
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 3, 2017, 6:34pm UTC](https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451/6 "2017-01-03T18:34:26Z")

</div>

I think it should be:

```auto
{
   "processors":[
      {
         "grok":{
            "field":"message",
            "patterns":[
               "%{STATUS:status};\\s+%{WORD:service};\\s+%{MSG:message}\\|\\s+%{WORD:key01}=%{NUMBER:value01}.*",
               "%{STATUS:status};\\s+%{WORD:service};\\s+%{GREEDYDATA:message}"
            ],
            "pattern_definitions":{
               "STATUS":"\\d+",
               "MSG":".+?"
            }
         }
      },
      {
         "grok":{
            "field":"source",
            "patterns":[
               ".+?%{TIMESTAMP:timestamp}.+"
            ],
            "pattern_definitions":{
               "TIMESTAMP":"[0-9]+"
            }
         }
      },
      {
         "date":{
            "field":"timestamp",
            "formats":[
               "UNIX_MS"
            ]
         }
      }
   ]
}

```

---

<div class="post-metadata">

**Author:** ![JamesFx](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@JamesFx](https://discuss.elastic.co/u/JamesFx)\
**Post date:** [January 3, 2017, 6:54pm UTC](https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451/7 "2017-01-03T18:54:01Z")

</div>

OMG... I am such a noob....

Sorry for that!

Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 31, 2017, 6:54pm UTC](https://discuss.elastic.co/t/elastic-ingest-with-multiple-grok-processors/70451/8 "2017-01-31T18:54:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.