Hi all,
I have a problem with the logstash grok configuration.
This is my current logstash configuration (logstash is running on a windows machine):
input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
}
}
output {
elasticsearch {
hosts => ["172.31.252.180:9200"]
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "(?m)%{SYSLOG5424LINE}" }
}
syslog_pri {
}
if !("_grokparsefailure" in [tags]) {
mutate {
replace => [ "message", "%{syslog5424_msg}" ]
replace => [ "timestamp", "%{syslog5424_ts}" ]
replace => [ "priority", "%{syslog5424_pri}" ]
replace => [ "program", "%{syslog5424_app}" ]
replace => [ "facility", "%{syslog_facility}" ]
replace => [ "severity", "%{syslog_severity}" ]
replace => [ "received_at", "%{@timestamp}" ]
}
mutate {
remove_field => [ "syslog5424_host", "syslog5424_msg", "syslog5424_ts", "syslog5424_pri", "syslog5424_app", "syslog5424_proc", "syslog5424_ver", "syslog_facility", "syslog_facility_code" , "syslog_severity", "syslog_severity_code" ]
}
}
}
}
I'm getting this kind of log with <> tag:
ClientPSV.GestTransPSV t=8D49D16FC840B37 tpsv=4395537650969499971 tpsvsigned=4395537650969499971 s=9000 m=2 msg='Socket UDP Error' exec=3139 <12>1 2017-05-17T10:01:03.216708Z SV-WEB051 psv 18820 - - ClientPSV.DecodeEVE t=4395537650969499971 ret=2020 msg=Data ora evento non valida134 <14>1 2017-05-17T10:01:03.218708Z SV-WEB051 psv 18820 - - End newEventID s=9000 m=2: t=4395537650969499971 stringResult=2020 eventid=0220 <11>1 2017-05-17T10:01:03.220708Z SV-WEB051 psvsched 19660 - - AamsRequestManager.openEventToAAMS t=8D49D17139C0A8F tpsv=4395537650969499971 platform=8 eventid=1575 pal=1702600500 ret=2020 msg='Event open failed' exec=18230 <11>1 2017-05-17T10:01:03.221709Z SV-WEB051 psvsched 19660 - - AamsRequestManager.openEventToAAMS t=8D49D17139C0A8F platform=8 eventid=1575 pal=1702600500 msg='Unexpected error'
It seems that multiple log are on the same record. How can I fix this?
Many Thanks
pjanzen
(Paul Janzen)
May 17, 2017, 11:00am
2
usally the <12> in the syslog lines are the prio's of that line. Your data looks mangled, now this could be due to copy & paste but I would except a line like this:
<11>1 2017-05-17T10:01:03.220708Z SV-WEB051 psvsched 19660 - - AamsRequestManager.openEventToAAMS t=8D49D17139C0A8F tpsv=4395537650969499971 platform=8 eventid=1575 pal=1702600500 ret=2020 msg='Event open failed' exec=18230
The above line is invalid syslog format but you could work around it.
you could also change your input to syslog { ... } that will give you a proper formatted message.
Thanks for your feedback.
This is the json version on Kibana:
{
"_index": "logstash-2017.05.17",
"_type": "syslog",
"_id": "AVwV8Wyx4Fgqgcyg76n_",
"_version": 1,
"_score": null,
"_source": {
"severity": "notice",
"program": "psvsched",
"message": "PsvSchedulerController.openKironEvents idevento=593553 apertura evento verso controller Kiron306 <15>1 2017-05-17T10:26:02.138585Z SV-WEB051 psvsched 19660 - - PSVKironEventScheduler idevento=593553 apertura kiron request message='<OpenEvent ID=\"593553\" ScheduleID=\"1702600498\" EventID=\"185\" IncludeForecast=\"false\" IncludeTricast=\"false\" IncludeReverseForecast=\"false\" IncludeReverseTricast=\"false\" />'145 <15>1 2017-05-17T10:26:02.734645Z SV-WEB051 psvsched 19660 - - PSVKironEventScheduler idevento=593553 apertura kiron response message='<Reply />'119 <14>1 2017-05-17T10:26:02.735645Z SV-WEB051 psvsched 19660 - - PSVKironEventScheduler idevento=593553 apertura kiron OK133 <14>1 2017-05-17T10:26:02.738645Z SV-WEB051 psvsched 19660 - - PSVKironEventScheduler idevento=593553 setStato=ApertoSuVGPlatorm OK\r",
"type": "syslog",
"priority": "14",
"@timestamp": "2017-05-17T10:26:02.529Z",
"port": 51809,
"received_at": "2017-05-17T10:26:02.529Z",
"@version": "1",
"host": "172.31.252.221",
"facility": "user-level",
"timestamp": "2017-05-17T10:26:02.138585Z"
},
"fields": {
"@timestamp": [
1495016762529
],
"received_at": [
1495016762529
],
"timestamp": [
1495016762138
]
},
"sort": [
1495016762529
]
}
I don't know why the format is invalid. My input is already syslog type, how can I workaround to get a proper formatted?
Thanks
I switch to
input {
syslog {
type => syslog
host => "172.31.252.180"
port => 5000
use_labels => false
}
}
but the result is still the same.
system
(system)
Closed
June 14, 2017, 11:33am
5
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.