Logstash 7 is not adding datestamp to index name

[kenwdelong (Ken)]

I just updated my stack from 6.7.0 -> 6.8.0 -> 7.4.0. Everything seems find EXCEPT logstash is putting all the events into an index called "logstash". I used to date-stamp the index, like "logstash-2019-10-24". I cannot figure out why, I've been pulling my hair out for 24 hours on this now.

I run just a single node in a docker container. All my config is here https://github.com/kenwdelong/elk-docker/tree/ELK-7.4.0, I didn't change anything from 6.7.0.

Any help would be appreciated!!!

[Badger]

It's a guess, but you might be seeing ILM in action. Instead of rolling over daily indexes it rolls over every 50 GB or 30 days. The elasticsearch output has options to manage or disable it.

[kenwdelong (Ken)]

Thanks for the suggestion. I disabled ILM explicitly in elasticsearch.yml, but to no avail.

However, I did find this:

[2019-10-25T00:35:14,703][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.10.25", :_type=>"_doc", :routing=>nil}, #LogStash::Event:0x36e64998], :response=>{"index"=>{"_index"=>"logstash-2019.10.25", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"The [default] mapping cannot be updated on index [logstash-2019.10.25]: defaults mappings are not useful anymore now that indices can have at most one type."}}}}

I'm not sure how to fix this yet but it looks like a clue.

[kenwdelong (Ken)]

I finally figured this out. I had an old template installed called "logstash" which still had the "defaults" key in it. Apparently ES 7 could not create the index because of the "defaults", so somehow it "fell back" to just "logstash". Also, I had some conflicts where ELK was making assumptions about field types (due to ECS) that broke stuff.

Now I just need to see if I can re-index the data languishing in "logstash" into a proper index...