Kibana stops displaying data at 12:00 AM every morning

Ever since we upgraded our production environment to R6.6.1 Kibana stops presenting data at 12:00 AM every morning and restarting the Kibana service doesn't resolve the issue. Documents are still being indexed into Elasticsearch and we are still able to search our repositories using in-house developed tools and get data back.

Restarting both the Logstash & Elasticsearch services on the production server resolves the issue but this definitely isn't ideal and our lower environments aren't exhibiting this behavior.

TIA,
Bill Youngman

Can you explain what you mean by "Kibana stops presenting data"? Is this in discover, or on a dashboard, or something else?

Lukas,

I've attached a screen shot of what I'm talking about

You'll notice that the last record displayed is at "March 20th 2019, 23:59:58.646" - this represents the time of the log entry. The @timestamp for this entry is "March 21st 2019, 00:00:01.053"

Here is a query that I just ran against one of our patterns followed by the return set:

POST requestlogv2-*/_search
{
"sort": [
{
"myTime": {
"order": "desc"
}
}
],
"_source": ["myTime","myUUID","myTID", "myMessageType"],
"query": {
"bool": {

"filter": [
  {
    "range": {
      "myTime": {
        "gte": "now-24h"
      }
    }
  }
]
}

}
}

1 sample return:

"hits" : [
  {
    "_index" : "requestlogv2-2019.03.21",
    "_type" : "request",
    "_id" : "__pnnmkBj6m6l96Y3nGG",
    "_score" : null,
    "_source" : {
      "myMessageType" : [
        "QuotationRequest"
      ],
      "myUUID" : "cb664563-4b8d-11e9-a581-0eacebdd395a",
      "myTime" : "2019-03-21T03:59:58.646Z",
      "myTID" : [
        "8154485741318828"
      ]
    },
    "sort" : [
      1553140798646
    ]

Thanks,
Bill

Figured out what the issue is and there is something invalid with our custom time field. When I changed and index pattern in a lower environment to use the @timestamp kibana started displaying up to date date.

Now the issue is when I try and delete an existing index pattern in production I'm getting this error--

blocked by: [FORBIDDEN/8/index write (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/8/index write (api)];

My research led to articles about not enough disk space or memory available on the machine to perform the write operation but we just added 1 TB to that machine so it's not that and the Elasticsearch is only using about 50% of the JVM Heap.

Also when I go to the Advanced Tab under Kibana and try to modify the settings I'm getting

Unable to update UI setting

Request failed with status code: 403

I reinstalled Kibana to no avail

Update--

The date that we are trying to use is in the following log format - 2019-03-22 16:28:47,916

The logstash filter transformation that we have declared is this:

date
{
	
        match => [ "myTime", "yyyy-MM-dd HH:mm:ss,SSS" ]
        target => "myTime"            
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.