Logstash mutate not replacing varible


(Ed) #1

I am on 6.2.4 and other variables seem to be working fine but my log_message is not

Anyone have a clue why logstash is ignoring %{log_message} ?

{

"_index": "dbinfra-2018.05.11",
"_type": "logs",
"_id": "AWNP9cpRpKvosm9xCQSk",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2018-05-11T16:08:03.564Z",
"@version": "1",
"filename": "alert_DDWOP1.log",
"path": "/oracle/product/diag/rdbms/ddwop/DDWOP1/trace/alert_DDWOP1.log",
"INSTANCE_NAME": "DDWOP1",
"tags": [
"multiline",
"_grokparsefailure",
"_dateparsefailure"
],
"ALERTLOG_FILE": "alert_DDWOP1",
"oradb_status": "running",
"DBNAME": "ddwop",
"host": "hd1mrc15na",
"dst_index": "dbinfra",
"type": "alertlog",
> "message": "%{log_message}"
},
"fields": {
"@timestamp": [
1526054883564
]
},
"sort": [
1526054883564
]
}


Logstash

Extract the date and the rest from the message

grok {
match => [ "message","%{DAY:day} %{MONTH:month} %{MONTHDAY:monthday} %{TIME:time} %{YEAR:year}(?<log_message>.*$)" ]
}
grok {
match => [ "path" , "/oracle/product/diag/rdbms/%{WORD:DBNAME}/%{WORD:INSTANCE_NAME}/trace/%{WORD:ALERTLOG_FILE}" ]
}

mutate {
add_field => {
"timestamp" => "%{year} %{month} %{monthday} %{time}"
}
}

replace the timestamp by the one coming from the alert.log

date {
locale => "en"
match => [ "timestamp" , "yyyy MMM dd HH:mm:ss" ]
}

replace the message (remove the date)

mutate { replace => [ "message", "%{log_message}" ] }


(Paris Mermigkas) #2

In your example there is a _grokparsefailure tag, meaning it failed to apply the first grok.
So the log_message field is never created, and as such the last replace fails to interpolate the variable and just inserts it as a literal string.

You should probably check why the initial grok fails in the first place.


(Ed) #3

Ah that would make sense, I found out the person who wrote that config and some other pieces I did not post just copy and pasted it from a blog. I went and re-wrote the whole thing, as there were lots of unneeded or meaningless mutates.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.