Error parsing syslog input message in Logstash

Elastic Stack Logstash
1

Hi to all,

I'm trying to parse a syslog message (coming from vmWare Log Insight) to obtain additional fields added inside the message body string, and then send them in json format to a kafka broker.

This is an example message obtained without filter in config file (only syslog input and output on text file with logstash):

{"@timestamp":"2019-03-15T10:01:28.978Z","@version":"1","message":"<14>1 2019-03-15T09:52:13.88Z hostname.fqdn.com - - - [Originator@6876 filepath="/path/logs/log.log" application_type="java" acronimo="xxxx0"] 338796f0-9e99-444a-b955-bba90512db62 2019-03-14 15:18:53,080 log messate text etc...","priority":0,"tags":["_grokparsefailure_sysloginput"],"severity_label":"Emergency","host":"1.1.1.1","facility_label":"kernel","severity":0,"facility":0}

Using it on https://grokdebug.herokuapp.com/ with filter:

^<%{NUMBER}>%{NUMBER}\s+%{TIMESTAMP_ISO8601:date}\s+%{HOSTNAME:src_hostname}\s+%{DATA}\s+%{DATA}\s+%{DATA}\s+[%{DATA}\s+filepath=\"%{DATA:src_filepath}\"\s+application_type=\"%{DATA:application_type}\"\s+acronimo=\"%{DATA:acronimo}\"]\s(?:%{UUID:loginsight_id})?%{GREEDYDATA:syslog_message}$

This is the result:
{
"NUMBER": [
[
"14",
"1"
]
],
"BASE10NUM": [
[
"14",
"1"
]
],
"date": [
[
"2019-03-15T09:52:13.88Z"
]
],
"YEAR": [
[
"2019"
]
],
"MONTHNUM": [
[
"03"
]
],
"MONTHDAY": [
[
"15"
]
],
"HOUR": [
[
"09",
null
]
],
"MINUTE": [
[
"52",
null
]
],
"SECOND": [
[
"13.88"
]
],
"ISO8601_TIMEZONE": [
[
"Z"
]
],
"src_hostname": [
[
"hostname.fqdn.com"
]
],
"DATA": [
[
"-",
"-",
"-",
"Originator@6876"
]
],
"src_filepath": [
[
"/path/logs/log.log"
]
],
"application_type": [
[
"java"
]
],
"acronimo": [
[
"xxxx0"
]
],
"loginsight_id": [
[
"338796f0-9e99-444a-b955-bba90512db62"
]
],
"syslog_message": [
[
" 2019-03-14 15:18:53,080 log messate text etc..."
]
]
}

I need to keep syslog_message, acronimo, application_type and src_hostname

This is the pipeline configured in logstash

input {
  syslog {
    port => 1514
  }
}


filter {
	grok {
		match => {
			"message" => '^<%{NUMBER}>%{NUMBER}\s+%{TIMESTAMP_ISO8601:date}\s+%{HOSTNAME:src_hostname}\s+%{DATA}\s+%{DATA}\s+%{DATA}\s+\[%{DATA}\s+filepath=\\\"%{DATA:src_filepath}\\\"\s+application_type=\\\"%{DATA:application_type}\\\"\s+acronimo=\\\"%{DATA:acronimo}\\\"\]\s(?:%{UUID:loginsight_id})?%{GREEDYDATA:syslog_message}$'
		}
	}
}

output {
  kafka {
    codec => json
    topic_id => "xxxx0"
  }
  stdout {
    codec => "rubydebug"
  }
}

This is the result on stdout:

"priority" => 0,
"@timestamp" => 2019-03-15T10:01:28.978Z
}
{
"message" => "<14>1 2019-03-15T09:52:13.88Z hostname.fqdn.com - - - [Originator@6876 filepath=\"/path/logs/log.log\" application_type=\"java\" acronimo=\"xxxx0\"] 338796f0-9e99-444a-b955-bba90512db62 2019-03-14 15:18:53,080 log messate text etc...",
"@version" => "1",
"facility_label" => "kernel",
"host" => "1.1.1.1",
"severity" => 0,
"facility" => 0,
"severity_label" => "Emergency",
"tags" => [
[0] "_grokparsefailure_sysloginput",
[1] "_grokparsefailure"
],

any idea?
I'm using logstash v6.6.1 with java version "1.8.0_162"
Regards

Luca

2

For the message shown in your post you do not need all the backslashes.

'^<%{NUMBER}>%{NUMBER}\s+%{TIMESTAMP_ISO8601:date}\s+%{HOSTNAME:src_hostname}\s+%{DATA}\s+%{DATA}\s+%{DATA}\s+\[%{DATA}\s+filepath="%{DATA:src_filepath}"\s+application_type="%{DATA:application_type}"\s+acronimo="%{DATA:acronimo}"\]\s(?:%{UUID:loginsight_id})?%{GREEDYDATA:syslog_message}$'
3

It works thanks!!!!!!
On logstash I means
but not on https://grokdebug.herokuapp.com/. :frowning:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.