I'm preparing an upgrade of ELK for a customer, moving from 6.2.0 to 7.4.1. The upgrade of ELK to 6.8.4 worked fine, however after upgrading to 7.4.1 logstash no longer sends data to ES.
ES and Kibana are both on 7.4.1 and seem to be working fine.
I ran a debug of both versions...
Logstash 6.8.4:
[2019-11-06T15:32:02,187][DEBUG][logstash.filters.grok ] Running grok filter {:event=>#<LogStash::Event:0x67d8770a>}
[2019-11-06T15:32:02,188][DEBUG][logstash.filters.grok ] Event now: {:event=>#<LogStash::Event:0x67d8770a>}
[2019-11-06T15:32:02,189][DEBUG][logstash.pipeline ] output received {"event"=> ...
Logstash 7.4.1 with same config/data
[2019-11-06T16:15:26,393][DEBUG][logstash.filters.grok ][main] Running grok filter {:event=>#<LogStash::Event:0x5d46d9de>}
[2019-11-06T16:15:26,394][DEBUG][logstash.filters.grok ][main] Event now: {:event=>#<LogStash::Event:0x5d46d9de>}
Logstash 6.8.4 has a pipeline/output entry after the grok filters, however with 7.4.1 no pipeline/output is shown in the debug output.
My config:
input {
file {
path => "E:/syslog-data/SyslogCatchAll*.txt"
type => "syslog"
sincedb_path => "E:/elk-data/logstash/sincedb/SyslogCatchAll.idx"
}
}
filter {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:syslog_timestamp}\t%{WORD:syslog_facility}\.%{WORD:syslog_level}\t%{IPV4:syslog_host_address}\t%{GREEDYDATA:syslog_message}"
}
}
if [syslog_host_address] == "10.255.255.2" {
mutate {
add_field => {
"syslog_srcdevice" => "fw"
}
}
kv {
source => "syslog_message"
prefix => "fw_"
}
}
if [syslog_host_address] in ["172.16.100.250","172.16.100.251","172.16.100.252","172.16.100.253"] {
mutate {
add_field => {
"syslog_srcdevice" => "ap"
}
gsub => [
"syslog_message", "\[", "(",
"syslog_message", "\]", ")"
]
}
grok {
match => {
"syslog_message" => "\{%{GREEDYDATA:ap_message}\}%{GREEDYDATA} Reason: %{GREEDYDATA:ap_reason}"
}
}
kv {
source => "ap_message"
field_split => " ,"
value_split => ":"
trim_key => "\""
trim_value => "\""
prefix => "ap_"
}
}
}
output {
elasticsearch {
hosts => ["10.0.0.1:9200"]
}
}
I did not see any breaking changes which might explain this behaviour.
Did I miss something? Do I need to add something to the elastic search output config with version 7+ ?