Kibana stops displaying data at 12:00 AM every morning

[byoungman (Bill Youngman)]

Ever since we upgraded our production environment to R6.6.1 Kibana stops presenting data at 12:00 AM every morning and restarting the Kibana service doesn't resolve the issue. Documents are still being indexed into Elasticsearch and we are still able to search our repositories using in-house developed tools and get data back.

Restarting both the Logstash & Elasticsearch services on the production server resolves the issue but this definitely isn't ideal and our lower environments aren't exhibiting this behavior.

TIA,
Bill Youngman

[lukas (Lukas Olson)]

Can you explain what you mean by "Kibana stops presenting data"? Is this in discover, or on a dashboard, or something else?

[byoungman (Bill Youngman)]

Lukas,

I've attached a screen shot of what I'm talking about

You'll notice that the last record displayed is at "March 20th 2019, 23:59:58.646" - this represents the time of the log entry. The @timestamp for this entry is "March 21st 2019, 00:00:01.053"

Here is a query that I just ran against one of our patterns followed by the return set:

POST requestlogv2-*/_search
{
"sort": [
{
"myTime": {
"order": "desc"
}
}
],
"_source": ["myTime","myUUID","myTID", "myMessageType"],
"query": {
"bool": {

"filter": [
  {
    "range": {
      "myTime": {
        "gte": "now-24h"
      }
    }
  }
]
}

}
}

1 sample return:

"hits" : [
  {
    "_index" : "requestlogv2-2019.03.21",
    "_type" : "request",
    "_id" : "__pnnmkBj6m6l96Y3nGG",
    "_score" : null,
    "_source" : {
      "myMessageType" : [
        "QuotationRequest"
      ],
      "myUUID" : "cb664563-4b8d-11e9-a581-0eacebdd395a",
      "myTime" : "2019-03-21T03:59:58.646Z",
      "myTID" : [
        "8154485741318828"
      ]
    },
    "sort" : [
      1553140798646
    ]

Thanks,
Bill

[byoungman (Bill Youngman)]

Figured out what the issue is and there is something invalid with our custom time field. When I changed and index pattern in a lower environment to use the @timestamp kibana started displaying up to date date.

Now the issue is when I try and delete an existing index pattern in production I'm getting this error--

blocked by: [FORBIDDEN/8/index write (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/8/index write (api)];

My research led to articles about not enough disk space or memory available on the machine to perform the write operation but we just added 1 TB to that machine so it's not that and the Elasticsearch is only using about 50% of the JVM Heap.

Also when I go to the Advanced Tab under Kibana and try to modify the settings I'm getting

Unable to update UI setting

Request failed with status code: 403

I reinstalled Kibana to no avail

[byoungman (Bill Youngman)]

Update--

The date that we are trying to use is in the following log format - 2019-03-22 16:28:47,916

The logstash filter transformation that we have declared is this:

date
{
	
        match => [ "myTime", "yyyy-MM-dd HH:mm:ss,SSS" ]
        target => "myTime"            
}