After upgrade from 6.8.4 to 7.4.1 no data is sent to ES

[MichelV]

I'm preparing an upgrade of ELK for a customer, moving from 6.2.0 to 7.4.1. The upgrade of ELK to 6.8.4 worked fine, however after upgrading to 7.4.1 logstash no longer sends data to ES.

ES and Kibana are both on 7.4.1 and seem to be working fine.

I ran a debug of both versions...

Logstash 6.8.4:

[2019-11-06T15:32:02,187][DEBUG][logstash.filters.grok    ] Running grok filter {:event=>#<LogStash::Event:0x67d8770a>}
[2019-11-06T15:32:02,188][DEBUG][logstash.filters.grok    ] Event now:  {:event=>#<LogStash::Event:0x67d8770a>}
[2019-11-06T15:32:02,189][DEBUG][logstash.pipeline        ] output received {"event"=> ...

Logstash 7.4.1 with same config/data

[2019-11-06T16:15:26,393][DEBUG][logstash.filters.grok    ][main] Running grok filter {:event=>#<LogStash::Event:0x5d46d9de>}
[2019-11-06T16:15:26,394][DEBUG][logstash.filters.grok    ][main] Event now:  {:event=>#<LogStash::Event:0x5d46d9de>}

Logstash 6.8.4 has a pipeline/output entry after the grok filters, however with 7.4.1 no pipeline/output is shown in the debug output.

My config:

input {
	file {
		path => "E:/syslog-data/SyslogCatchAll*.txt"
		type => "syslog"
		sincedb_path => "E:/elk-data/logstash/sincedb/SyslogCatchAll.idx"
	}
}

filter {
	grok {
		match => {
			"message" => "%{TIMESTAMP_ISO8601:syslog_timestamp}\t%{WORD:syslog_facility}\.%{WORD:syslog_level}\t%{IPV4:syslog_host_address}\t%{GREEDYDATA:syslog_message}"
		}
	}

	if [syslog_host_address] == "10.255.255.2" {
		mutate {
			add_field => {
				"syslog_srcdevice" => "fw"
			}
		}
		kv {
			source => "syslog_message" 
			prefix => "fw_"
		}
	}

	if [syslog_host_address] in ["172.16.100.250","172.16.100.251","172.16.100.252","172.16.100.253"] {
		mutate {
			add_field => {
				"syslog_srcdevice" => "ap"
			}
			gsub => [
				"syslog_message", "\[", "(",
				"syslog_message", "\]", ")"
			]
		}
		
		grok {
			match => {
				"syslog_message" => "\{%{GREEDYDATA:ap_message}\}%{GREEDYDATA} Reason: %{GREEDYDATA:ap_reason}"
			}
		}

		kv {
			source => "ap_message"
			field_split => " ,"
			value_split => ":"
			trim_key => "\""
			trim_value => "\""
			prefix => "ap_"
		}
	}
}

output {
	elasticsearch { 
		hosts => ["10.0.0.1:9200"]
	}
}

I did not see any breaking changes which might explain this behaviour.

Did I miss something? Do I need to add something to the elastic search output config with version 7+ ?

[MichelV]

Nobody?

I executed the same upgrade on a totally different machine (customer VM), but got the same result.

Has there been any change to the Elasticsearch output plugin I'm missing? Or any changes to requirements to connect to Elasticsearch?

[MichelV]

After I deleted the ES database, it worked again. Events via Logstash are added to ES.
So a clean database works fine, an upgraded database not.

Sounds like something isn't upgraded (correctly) during the upgrade proces? Now how to figure out what?

How am i going to tell our customer we will have to delete a 4TB database to get the upgrade working...

[MichelV]

Continuing my monologue....

I found out that after upgrading Elasticsearch from 6.8.4 to 7.4.2 the logstash template gets corrupted (??)

Logstash template in ES 6.8.4:

{"logstash":{"order":0,"version":60001,"index_patterns":["logstash-"],"settings":{"index":{"refresh_interval":"5s"}},"mappings":{"default":{"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"type":"text","norms":false}}},{"string_fields":{"match":"","match_mapping_type":"string","mapping":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"geoip":{"dynamic":true,"properties":{"ip":{"type":"ip"},"location":{"type":"geo_point"},"latitude":{"type":"half_float"},"longitude":{"type":"half_float"}}}}}},"aliases":{}}}

After upgrade to ES 7.4.2:

{"logstash":{"order":0,"version":60001,"index_patterns":["logstash-*"],"settings":{"index":{"refresh_interval":"5s"}},"mappings":{},"aliases":{}}}

Since there is a logstash template with the same version nr, logstash skips installing/updating the template:

[2019-11-26T15:50:53,939][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, [...snip...]
[2019-11-26T15:50:54,079][DEBUG][logstash.outputs.elasticsearch][main] Found existing Elasticsearch template. Skipping template management {:name=>"logstash"}

I then deleted the logstash template:

Invoke-RestMethod -Uri "http://localhost:9200/_template/logstash?pretty" -Method Delete

Logstash (version 7.4.2) recreates the correct logstash template... but unfortunately this doesn't fix my issue. There is still no data is stored in ES by the new Logstash 7.4.2.

Also, I don't see "[DEBUG][logstash.pipeline] output received" in the debug logging with 7.4.2, which I did see with 6.8.4.

[MichelV]

So... while logstash indices were formatted as logstash-yyyy.mm.dd, when I upgraded to 7.4.2 the index was just logstash.

In Kibana I had a logstash-* filter (including the dash) which offcourse would exclude the new index. This explains why I did not see anything in Kibana...

Why did this change? Is this a configurable setting (yes, I'm still a noob with ELK)?

The absence of any "[DEBUG][logstash.pipeline] output received" messages did not help aswell...

This is frustrating...

[Badger]

I think this is related to ILM. However, "logstash" is supposed to be the rollover alias, which should get aliased to a different index name.

[MichelV]

You were right about ILM. After disabling ILMin the config, the index is created as before (logstash-yyyy.mm.dd).

output {
    elasticsearch { 
        hosts => ["localhost:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
        ilm_enabled => false
    }
}