Hi guys,
I'm trying to define two different grok statements in my pipeline as I can receive more than 1 message pattern log.
This is my pipeline simulation:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{SYSLOGTIMESTAMP} %{GREEDYDATA} %{LOGLEVEL:loglevel} \\[%{DATA:comp2}\\] %{GREEDYDATA:tibMsgCode} \\[%{DATA:action}\\] %{GREEDYDATA:msg}",
"%{GREEDYDATA:msg}"
],
"on_failure": [
{
"set": {
"field": "_index",
"value": "failed-{{ _index }}"
}
},
{
"set": {
"field": "error",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}
}
]
},
"docs": [
{
"_source": {
"message": "2016 Nov 04 11:08:10:049 GMT +0000 : Keystore inited."
},
"_index": "index"
}
]
}
And the output is:
{
"docs": [
{
"doc": {
"_id": "_id",
"_type": "_type",
"_index": "index",
"_source": {
"message": "2016 Nov 04 11:08:10:049 GMT +0000 : Keystore inited."
},
"_ingest": {
"timestamp": "2017-01-23T22:32:30.891+0000"
}
}
}
]
}
Correct if I'm wrong but it seems like is not picking any pattern defined and also is not triggering the on_failure code, what could be wrong? If I change the order of the grok statements it throw something like this:
{
"docs": [
{
"doc": {
"_id": "_id",
"_type": "_type",
"_index": "index",
"_source": {
"msg": "2016 Nov 04 11:08:10:049 GMT +0000 : Keystore inited.",
"message": "2016 Nov 04 11:08:10:049 GMT +0000 : Keystore inited."
},
"_ingest": {
"timestamp": "2017-01-23T22:44:34.412+0000"
}
}
}
]
Which seems to be working, but then if I try a different string like:
"message": "2016 Nov 04 11:16:21:945 GMT +0000 Info [Application] AESDKJ-0000 [http-bio-8443-exec-68] [Query log files] The log file dummyApp has been queried by dummyUser."
It won't match the desired grok statement.
Thanks in advance!