How to use more than 1 pattern in pipelines?

[kikecueva (Enrique Cuevas)]

Hi guys,

I'm trying to define two different grok statements in my pipeline as I can receive more than 1 message pattern log.
This is my pipeline simulation:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{SYSLOGTIMESTAMP} %{GREEDYDATA} %{LOGLEVEL:loglevel} \\[%{DATA:comp2}\\] %{GREEDYDATA:tibMsgCode} \\[%{DATA:action}\\] %{GREEDYDATA:msg}",
            "%{GREEDYDATA:msg}"
          ],
          "on_failure": [
            {
              "set": {
                "field": "_index",
                "value": "failed-{{ _index }}"
              }
            },
            {
              "set": {
                "field": "error",
                "value": "{{ _ingest.on_failure_message }}"
              }
            }
          ]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "2016 Nov 04 11:08:10:049 GMT +0000 : Keystore inited."
      },
      "_index": "index"
    }
  ]
}

And the output is:

{
  "docs": [
    {
      "doc": {
        "_id": "_id",
        "_type": "_type",
        "_index": "index",
        "_source": {
          "message": "2016 Nov 04 11:08:10:049 GMT +0000 : Keystore inited."
        },
        "_ingest": {
          "timestamp": "2017-01-23T22:32:30.891+0000"
        }
      }
    }
  ]
}

Correct if I'm wrong but it seems like is not picking any pattern defined and also is not triggering the on_failure code, what could be wrong? If I change the order of the grok statements it throw something like this:

{
  "docs": [
    {
      "doc": {
        "_id": "_id",
        "_type": "_type",
        "_index": "index",
        "_source": {
          "msg": "2016 Nov 04 11:08:10:049 GMT +0000 : Keystore inited.",
          "message": "2016 Nov 04 11:08:10:049 GMT +0000 : Keystore inited."
        },
        "_ingest": {
          "timestamp": "2017-01-23T22:44:34.412+0000"
        }
      }
    }
  ]

Which seems to be working, but then if I try a different string like:

"message": "2016 Nov 04 11:16:21:945 GMT +0000 Info [Application] AESDKJ-0000 [http-bio-8443-exec-68] [Query log files] The log file dummyApp has been queried by dummyUser."

It won't match the desired grok statement.

Thanks in advance!

[kikecueva (Enrique Cuevas)]

By the way I'm using ELK 5.1