Logstash GROK parse for syslog

[engineering87]

Hi all,
I have a problem with the logstash grok configuration.
This is my current logstash configuration (logstash is running on a windows machine):

input {
  tcp {
    port => 5000
    type => syslog
  }
  udp {
    port => 5000
    type => syslog
  }
}

output {  
  elasticsearch {
    hosts => ["172.31.252.180:9200"]
  }
}

filter {
	if [type] == "syslog" {
		grok {
			match => { "message" => "(?m)%{SYSLOG5424LINE}" }
		}
		syslog_pri {

		}
		if !("_grokparsefailure" in [tags]) {
			mutate {
				replace => [ "message", "%{syslog5424_msg}" ]
				replace => [ "timestamp", "%{syslog5424_ts}" ]
				replace => [ "priority", "%{syslog5424_pri}" ]
				replace => [ "program", "%{syslog5424_app}" ]
				replace => [ "facility", "%{syslog_facility}" ]
				replace => [ "severity", "%{syslog_severity}" ]
				replace => [ "received_at", "%{@timestamp}" ]
			}
			mutate {
				remove_field => [ "syslog5424_host", "syslog5424_msg", "syslog5424_ts", "syslog5424_pri", "syslog5424_app", "syslog5424_proc", "syslog5424_ver", "syslog_facility", "syslog_facility_code" , "syslog_severity", "syslog_severity_code" ]
			}
		}
	}
}

I'm getting this kind of log with <> tag:

ClientPSV.GestTransPSV t=8D49D16FC840B37 tpsv=4395537650969499971 tpsvsigned=4395537650969499971 s=9000 m=2 msg='Socket UDP Error' exec=3139 <12>1 2017-05-17T10:01:03.216708Z SV-WEB051 psv 18820 - - ClientPSV.DecodeEVE t=4395537650969499971 ret=2020 msg=Data ora evento non valida134 <14>1 2017-05-17T10:01:03.218708Z SV-WEB051 psv 18820 - - End newEventID s=9000 m=2: t=4395537650969499971 stringResult=2020 eventid=0220 <11>1 2017-05-17T10:01:03.220708Z SV-WEB051 psvsched 19660 - - AamsRequestManager.openEventToAAMS t=8D49D17139C0A8F tpsv=4395537650969499971 platform=8 eventid=1575 pal=1702600500 ret=2020 msg='Event open failed' exec=18230 <11>1 2017-05-17T10:01:03.221709Z SV-WEB051 psvsched 19660 - - AamsRequestManager.openEventToAAMS t=8D49D17139C0A8F platform=8 eventid=1575 pal=1702600500 msg='Unexpected error'

It seems that multiple log are on the same record. How can I fix this?

Many Thanks

[pjanzen (Paul Janzen)]

usally the <12> in the syslog lines are the prio's of that line. Your data looks mangled, now this could be due to copy & paste but I would except a line like this:

<11>1 2017-05-17T10:01:03.220708Z SV-WEB051 psvsched 19660 - - AamsRequestManager.openEventToAAMS t=8D49D17139C0A8F tpsv=4395537650969499971 platform=8 eventid=1575 pal=1702600500 ret=2020 msg='Event open failed' exec=18230

The above line is invalid syslog format but you could work around it.

you could also change your input to syslog { ... } that will give you a proper formatted message.

[engineering87]

Thanks for your feedback.
This is the json version on Kibana:

{
  "_index": "logstash-2017.05.17",
  "_type": "syslog",
  "_id": "AVwV8Wyx4Fgqgcyg76n_",
  "_version": 1,
  "_score": null,
  "_source": {
    "severity": "notice",
    "program": "psvsched",
    "message": "PsvSchedulerController.openKironEvents idevento=593553 apertura evento verso controller Kiron306 <15>1 2017-05-17T10:26:02.138585Z SV-WEB051 psvsched 19660 - - PSVKironEventScheduler idevento=593553 apertura kiron request message='<OpenEvent ID=\"593553\" ScheduleID=\"1702600498\" EventID=\"185\" IncludeForecast=\"false\" IncludeTricast=\"false\" IncludeReverseForecast=\"false\" IncludeReverseTricast=\"false\" />'145 <15>1 2017-05-17T10:26:02.734645Z SV-WEB051 psvsched 19660 - - PSVKironEventScheduler idevento=593553 apertura kiron response message='<Reply />'119 <14>1 2017-05-17T10:26:02.735645Z SV-WEB051 psvsched 19660 - - PSVKironEventScheduler idevento=593553 apertura kiron OK133 <14>1 2017-05-17T10:26:02.738645Z SV-WEB051 psvsched 19660 - - PSVKironEventScheduler idevento=593553 setStato=ApertoSuVGPlatorm OK\r",
    "type": "syslog",
    "priority": "14",
    "@timestamp": "2017-05-17T10:26:02.529Z",
    "port": 51809,
    "received_at": "2017-05-17T10:26:02.529Z",
    "@version": "1",
    "host": "172.31.252.221",
    "facility": "user-level",
    "timestamp": "2017-05-17T10:26:02.138585Z"
  },
  "fields": {
    "@timestamp": [
      1495016762529
    ],
    "received_at": [
      1495016762529
    ],
    "timestamp": [
      1495016762138
    ]
  },
  "sort": [
    1495016762529
  ]
}

I don't know why the format is invalid. My input is already syslog type, how can I workaround to get a proper formatted?

Thanks

[engineering87]

I switch to

input {
  syslog {
    type => syslog
    host => "172.31.252.180"
    port => 5000
	use_labels => false
  }
}

but the result is still the same.