Bit of a head scratcher here, as I wasn't sure which area to post this in.
Up until May 28th, 11:59:50 PM I was able to view data from the only filebeat server running version 6.7.1 (the rest are running 7.0.1). For weeks, it's been working just fine. I use it to parse suricata data from our pfsense box. May 29th came and a new filebeat-6.7.1-2019.05.29 index was created. It's receiving data into the index as debug log files from filebeat show the events as being successfully published and I can see the index growing in document/filesize via the Index Management area in Kibana. However, when I try to view any of the data via "Discover" area in Kibana, none of it is shown. If I do a view over the last 24hrs, there is a sharp division when the new index is created for May 29th - basically, nothing is listed. Data for filebeat version 7.0.1 from the other hosts is viewable.
I've also restarted all suricata processes on the interfaces we monitor. I thought that the nightly update for their rules might have caused some issue, but that runs at 12:30am and were that related I would have expected to see data up until that time. But it cuts off right at 12am.
Oddly, though, all the dashboards and their visualizations still work so it seems that the data is there and accessible, but just not viewable to search through in "Discover".
Has anyone else run across this? Filebeat says the data is being published, so I didn't post in that section. And Elasticsearch logs don't show anything unusual. It sort of crosses over between products so I apologize if this is not the section where this should have been posted.
Any ideas?