Ever since we upgraded our production environment to R6.6.1 Kibana stops presenting data at 12:00 AM every morning and restarting the Kibana service doesn't resolve the issue. Documents are still being indexed into Elasticsearch and we are still able to search our repositories using in-house developed tools and get data back.
Restarting both the Logstash & Elasticsearch services on the production server resolves the issue but this definitely isn't ideal and our lower environments aren't exhibiting this behavior.
You'll notice that the last record displayed is at "March 20th 2019, 23:59:58.646" - this represents the time of the log entry. The @timestamp for this entry is "March 21st 2019, 00:00:01.053"
Here is a query that I just ran against one of our patterns followed by the return set:
Figured out what the issue is and there is something invalid with our custom time field. When I changed and index pattern in a lower environment to use the @timestamp kibana started displaying up to date date.
Now the issue is when I try and delete an existing index pattern in production I'm getting this error--
My research led to articles about not enough disk space or memory available on the machine to perform the write operation but we just added 1 TB to that machine so it's not that and the Elasticsearch is only using about 50% of the JVM Heap.
Also when I go to the Advanced Tab under Kibana and try to modify the settings I'm getting
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.