Read this... And follow the steps
Where it's say to clean up that means completely remove any / All metricbeat on indices and or data streams
Gives you step by step...
And with the new datastreams logstash should look like this.
input {
beats {
port => 5044
}
}
output {
if [@metadata][pipeline] {
elasticsearch {
hosts => "http://localhost:9200"
pipeline => "%{[@metadata][pipeline]}"
user => "elastic"
password => "password"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
action => "create"
}
} else {
elasticsearch {
hosts => "http://localhost:9200"
user => "elastic"
password => "password"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
action => "create"
}
}
}
These are the steps / configuration that work / work for others.
Also, you're logstash config is not correct. You're not looking close at the elasticsearch output. You need to write to the data stream so it uses to correct mapping and then rolls over with the island. When you added the date to it, it wasn't doing that correctly.
Get that working then you can put Kafka back in.