How to parse Syslog messages

Elastic Stack Logstash
1

Need support in parsing Syslog messages

I want to parse the below syslog message.

<13> 172.0.0.2 Nov 9 09:53:53 172.0.0.2 1 2016-11-09T09:57:14.018719Z [interface] [172.0.0.2]
timestamp=2016-11-09 09:57:14 machine=DDoS pdomain=home in_pps_tot=29245

I know we need to use KV filter for parsing. I put lots of effort on making it work but I was unsuccessful. _grokparsefailure!!
Looking forward for the help from Elastic community.

The code I am currently working on is below:

input {
udp {
port => 8000
type => syslog
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "<%{POSINT:syslog_pri"> %{SYSLOGHOST:syslog_hostname} %{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:facility} %{SYSLOGTIMESTAMP:syslog_timestamp2} \[%{DATA:event_name}\] \[%{SYSLOGHOST:syslog_hostname}\] %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ] } }}}

syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
}
date {
match => [ "syslog_timestamp2", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ", "ISO8601" ]
}

kv{
source => "syslog_message"
remove_field => ["syslog_message"]
}

output {
elasticsearch { host => localhost
index => "dashboard"
}
stdout { codec => rubydebug }
}

2
  • There's an extra double quote right after "syslog_pri".
  • Don't use DATA to capture facility, use a more exact pattern like e.g. NUMBER instead.
  • Use TIMESTAMP_ISO8601 to match the second timestamp.

If it still doesn't work, start building the expression from the beginning, i.e. start with <%{POSINT:syslog_pri> and verify that that works, then add piece by piece until it stops working.

3

thank you !!!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.