Upgraded to 6.0.0, where is my latest data!?

[cosmo]

Hello

I upgraded from 5.6 to 6.0 yesterday.

I have a cron to download a daily log, then pass that into logstash to index all the events.
eg bin/logstash -f custom-config.yml < /logs/log-2017-11-15.log

I haven't changed the config, but logstash seems to have hidden my data somewhere.

Normally it would create an index logstash-2017.11.15 (the default pattern) but I can't find that index for yesterday.

I am using xpack with a basic license, I don't know if that could have effected anything?

I just installed xpack in logstash, and have now got the Logstash monitoring in Kibana, which says it has received and emitted about a million events, which is about right for the log file. But I can't for the life of me find them!

I have run the command manually with --debug but that doesn't tell me anything, and there don't seem to be any errors in /var/log/elasticsearch/elasticsearch.log

Have I missed some obvious step in the upgrade??

Thanks
Chris

[cosmo]

Also just to add, I have a custom PHP script that inserts documents straight into the same elasticsearch and that is still working fine (after upgrading the php elastic module version)

[cosmo]

I think I've figure this out.
I'm using the default logstash templates, but didn't read the part of the docs saying I now have to set template_overwrite = true

https://www.elastic.co/guide/en/logstash/current/upgrading-logstash-6.0.html
(also pull request to correct overwrite_template to template_overwrite)