Grok pattern for this syslog message for Logstash?

Elastic Stack Logstash
1

Hello

I have this syslog message which is ALMOST like the standard RFC3164 so the default syslog plugin should pick it up:

<134>1 2021-10-05T08:48:18Z MYSERVER iLO5 - - - XML logout: SomeUser - 1.2.3.4(DNS name not found).

I was reading the RFC and (this is offtopic), I honestly do not understand how to break down the 134; I know it is a bit representation of it being a emergency, critical, etc. but I dont seem to understand it.

The rest seems pretty straight forward.

Is that "1" after <134> messing everything up?

Thank you for the help

2

You can use the syslog pri filter to parse after you grok it if you needed to.

Also I do not know what that 1 is, never seen anything like that. You would need to go back to what is generating the log to figure that out.

Test Conf

input { generator { lines => ['{ "message": 134 }'] count => 1 codec => "json" } }
filter{
  syslog_pri { } 
}
output { stdout { codec => json } }

Output

{
    "@timestamp": "2021-10-05T10:52:35.412Z",
    "syslog_severity_code": 5,
    "syslog_facility_code": 1,
    "syslog_severity": "notice",
    "message": 134,
    "syslog_facility": "user-level"
}
3

The 1 after the syslog pri is the syslog protocol version.

The RFC 3164 is obsolete, you should look at the RFC 5424.

The version is described in this part of the RFC 5424 and the syslog pri calculation is explained in this part of the RFC.

As Aaron said, the syslog_pri filter you get you the syslog_facility and syslog_severity from the syslog_pri number.

4

There is something Im missing.

Ill post my current Logstash config


input {
syslog {
    timezone => "Europe/Madrid"
    port => "5062"
    type => "syslog"
}
}



filter
{
grok {
   match => { "message" => "%{SYSLOG5424PRI} %{DATE_US:DateILO} %{TIME:TimeiLO} %{GREEDYDATA:LogMessage}" }
 }

mutate {
		add_field => { "iLOTimestamp" => "%{DateILO} %{TimeiLO}"}	
} 
 
date {
match => ["iLOTimestamp", "MM/dd/yyyy HH:mm:ss"]
timezone => "UTC"
target => "iLOTimestamp"
}
 
 
}




output {
elasticsearch {
    hosts => ["localhost"]
    user => ["elastic"]
    password  => ["CENSORED"]
    ilm_enabled => true
    ilm_policy => "logs-ilp"
	index => "syslog-hpeilologs-%{+yyyy.MM}"
    }

stdout { codec => rubydebug }
}

Im using the date plugin because for some odd reason, my iLO doesnt show local time, instead it shows UTC time and Ive been trying to translate it but nothing so far (Thats another topic)

Now, it looks very difficult to what you are proposing and starting with the input. Could you please explain? Thank you.

5

I found a even better one:


filter
{
grok {
   match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:version} %{TIMESTAMP_ISO8601:DateILO} %{DATA:HostName} %{GREEDYDATA:LogMessage}" }
 }
}

The only thing that is not working is SYSLOG5424PRI; It doesnt split that initial value into Emergency, etc....

6

You have to add the above to do that parsing.

7

Where exactly? Just add it?

8

Yup....lol...It just works.....

I still do not understand Logstash at all on how it works.

1 Like
9

Looking way better

@timestamp
Oct 5, 2021 @ 16:28:18.011
	
@version
1
	
DateILO
Oct 5, 2021 @ 16:28:17.000
	
HostName
HOSTNAME
	
LogMessage
iLO5 - - - XML logout: user - 1.2.3.4(DNS name not found).
	
_id
c83aUHwBCddptw5j8GpB
	
_index
syslog-hpeilologs-2021.10
	
_score
 - 
	
_type
_doc
	
facility
0
	
facility_label
kernel
	
host
1.2.3.4
	
message
<134>1 2021-10-05T14:28:17Z HOSTNAME iLO5 - - - XML logout: user - 1.2.3.4(DNS name not found).
	
priority
0
	
severity
0
	
severity_label
Emergency
	
syslog5424_pri
134
	
syslog_facility
user-level
	
syslog_facility_code
1
	
syslog_severity
notice
	
syslog_severity_code
5
	
syslogversion
1
	
tags
_grokparsefailure_sysloginput
	
type
syslog

That being said, I see a facility_label and a severity_label that are different from syslog_facility and syslog_severity ; Why? How can I remove them? Or fix them?

Also I see "tags
_grokparsefailure_sysloginput" which is usually because something has failed. Any way to fix?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.