Messages grok parse failure

[mvelez83 (Manny Velez)]

Hello,

I'm very new to creating Grok filters and I was going to create one for /var/log/messages (For named). Here is my logstash configuration I attempted to use:

input {
redis {
host => "redis01.nyc.aevtech.net"
data_type => "list"
key => "dnsint"
password => "qwarsdw5243"
}
}
filter {
grok {
match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA}" }
}
}
output {
elasticsearch { hosts => ["es01.nyc.aevtech.net:9200"]
index => "logstash-%{+YYYY.MM.dd}" }
}

I can see the logs coming in but I also see the following tags with each log entry coming in for /var/log/messages:

tags: _grokparsefailure, _geoip_lookup_failure, _dateparsefailure

I'm not 100% sure what the issue is. Also, I tried using Automatic grok discovery to come up with this grok filter but as mentioned above this is very new to me. Thank you for any and all help in advanced and let me know if anything else is needed to help assist so I can send it as soon as possible.

Thank You

[mvelez83 (Manny Velez)]

Hello,

Here is some of the log entries I would like this grok filter to work for:

Jul 4 13:14:09 dns01 named[31894]: validating @0x7fadc0710810: sumo.external.zlb.scl3.mozilla.com A: no valid signature found
Jul 4 13:59:39 dns01 named[31894]: error (connection refused) resolving 'cdn-hls.movies.com.fpbns.net/A/IN': 192.20.123.10#53
Jul 4 14:39:15 dns01 named[31894]: dispatch 0x7fadc00f5f20: open_socket(0.0.0.0#8614) -> permission denied: continuing
Jul 4 15:27:09 dns01 named[31894]: validating @0x7fadc0710810: choices.truste.com A: no valid signature found
Jul 4 15:28:10 dns01 named[31894]: validating @0x7fadc067b940: choices-or.truste.com A: no valid signature found
Jul 4 15:38:39 dns01 named[31894]: validating @0x7fadc067b940: choices.truste.com A: no valid signature found
Jul 4 15:38:42 dns01 named[31894]: validating @0x7fadc0710810: choices-or.truste.com A: no valid signature found
Jul 4 17:42:18 dns01 named[31894]: dispatch 0x7fadc00f6540: open_socket(0.0.0.0#4321) -> permission denied: continuing
Jul 4 17:42:37 dns01 named[31894]: error (connection refused) resolving 'webdir.online.lync.com/A/IN': 8.26.204.25#53
Jul 4 19:01:15 dns01 named[31894]: dispatch 0x7fadc00f6b60: open_socket(0.0.0.0#8610) -> permission denied: continuing

[Nico-DF]

You're currently using SYSLOGBASE in your pattern.

If you check what it is (http://grokdebug.herokuapp.com/patterns#), you can find the following:

SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:

SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
PROG (?:[\w._/%-]+)
SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
SYSLOGHOST %{IPORHOST}
SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>

So basically, you see that your log line does not match this pattern. Hence the grokparsefailure

[mvelez83 (Manny Velez)]

Hello,

Thank you very much for your reply. So, based off of what you said I tried a new one and also passed it along the Grok Debugger and came up with the following:

%{SYSLOGBASE2} %{GREEDYDATA}

This produced this:

{
"SYSLOGBASE2": [
[
"Jul 5 04:54:18 dns01 systemd:"
]
],
"timestamp": [
[
"Jul 5 04:54:18"
]
],
"MONTH": [
[
"Jul"
]
],
"MONTHDAY": [
[
"5",
null
]
],
"TIME": [
[
"04:54:18"
]
],
"HOUR": [
[
"04",
null,
null
]
],
"MINUTE": [
[
"54",
null,
null
]
],
"SECOND": [
[
"18",
null
]
],
"timestamp8601": [
[
null
]
],
"YEAR": [
[
null
]
],
"MONTHNUM": [
[
null
]
],
"ISO8601_TIMEZONE": [
[
null
]
],
"SYSLOGFACILITY": [
[
null
]
],
"facility": [
[
null
]
],
"priority": [
[
null
]
],
"logsource": [
[
"dns01"
]
],
"IPORHOST": [
[
"dns01"
]
],
"HOSTNAME": [
[
"dns01"
]
],
"IP": [
[
null
]
],
"IPV6": [
[
null
]
],
"IPV4": [
[
null
]
],
"SYSLOGPROG": [
[
"systemd"
]
],
"program": [
[
"systemd"
]
],
"pid": [
[
null
]
],
"GREEDYDATA": [
[
"Starting Session 313 of user mvelez."
]
]
}

When I restarted logstash I can see the logs coming in with more fields and options compared to before but I'm still seeing the tags with the parse failures. Is it because it is not grabbing the full line in the pattern? Compared to before, I can grab much more data from the logs. I guess the question for me now is that although the above is a bit better will those tags be an issue?

Thank You!