Default index pattern goes missing

rs_ela · December 4, 2017, 2:30pm

Hi all. Anyone also having this issue where ocassionally the default index pattern is no longer default and Kibana needs user to "star" some pattern once again? It happens on my instance every once in a while. Can't come up with a stable repro, though, to be able to confirm it as a bug.

The closest I am to having a stable repro is as follows:

delete the default index (in my case it's "logstash-*")
recreate the index
set it as default
let ELK to work under load overnight
it might happen that the next morning none index pattern is reported as default (or it might occur within minutes)

I've been noticing this in ELK v5.6.x (Debian).

Anyone can confirm?

weltenwort · December 5, 2017, 4:02pm

Hi @rs_ela,

the default index pattern is persisted in the internal .kibana index (that name could be changed via the settings). Is it possible that there is more than just that Kibana instance writing to that index, e.g. a second instance or a script that changes Kibana settings? Does that only affect the default index pattern or are other settings like the "advanced settings" also affected?

rs_ela · December 6, 2017, 11:23am

Thanks for the feedback, @weltenwort.

There is only one Kibana.

Regarding scripts - there is none that would mess with ".kibana" index. There is one script that periodically deletes older "logstash-..." indexes to free disk space, but it only touches indexes with names beginning with "logstash-...". If that was the cause for the issue, they would come up at least daily.

Regarding "advanced settings" - no idea. Didn't notice anything behaving / looking differently, but did not check those settings, as well, so can't be 100% sure.

weltenwort · December 7, 2017, 5:23pm

I can only speculate about the causes, but it would be interesting to see whether there is a document with type: "config" in the .kibana index or whether it vanished. In case the problem occurs again, would you mind checking that and maybe posting the document content?

rs_ela · December 8, 2017, 8:31am

Sure, I can do that. I only want to mention that it can take some (a lot of?) time until the problem reoccurs.

Currently I can see "config" dokument on .kibana index, can you please confirm that this is what we're looking for?

  "config" : {
    "properties" : {
      "buildNum" : {
        "type" : "keyword"
      },
      "defaultIndex" : {
        "type" : "text",
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "discover:aggs:terms:size" : {
        "type" : "long"
      },
      "state:storeInSessionStorage" : {
        "type" : "boolean"
      }
    }
  },

weltenwort · December 11, 2017, 9:53am

This looks like the mapping definition of the config type. It's hard to tell without a reliable way to reproduce it. Would it be possible for you to post the result of GET .kibana/config/_search?

rs_ela · December 11, 2017, 12:17pm

Here we go:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 13,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.4.0",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15063,
          "defaultIndex" : "*",
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      },
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.4.1",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15104,
          "defaultIndex" : "*",
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      },
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.4.2",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15117,
          "defaultIndex" : "*",
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      },
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.4.3",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15122,
          "defaultIndex" : "logstash-*",
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      },
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.5.0",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15382,
          "defaultIndex" : "logstash-*",
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      },
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.5.1",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15405,
          "defaultIndex" : "logstash-*",
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      },
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.5.2",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15443,
          "defaultIndex" : "logstash-*",
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      },
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.6.0",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15523,
          "defaultIndex" : "logstash-*",
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      },
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.6.1",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15533,
          "defaultIndex" : "logstash-*",
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      },
      {
        "_index" : ".kibana",
        "_type" : "config",
        "_id" : "5.6.2",
        "_score" : 1.0,
        "_source" : {
          "buildNum" : 15543,
          "defaultIndex" : null,
          "discover:aggs:terms:size" : 20,
          "metaFields" : null
        }
      }
    ]
  }
}

weltenwort · December 12, 2017, 10:23am

So it looks like there is a config document for version 5.6.1 with the default index set to logstash-* and one for 5.6.2 without a default index. I assume you are running version 5.6.2 of Kibana right now? Does the state of the .kibana index you showed above correspond to a situation where the default index is not set?

rs_ela · December 12, 2017, 2:44pm

Nope I'm running Kibana 5.6.5.

Not sure what you mean by "correspond to a situation", but at the moment there are no issues with default indexes. I am still waiting for this issue to occur again.

weltenwort · December 13, 2017, 11:16am

I just noticed that the response was limited by the default response size of 10. Could you maybe perform the query again with

GET .kibana/config/_search
{ "size": 100 }

?

rs_ela · December 13, 2017, 12:15pm

Sure:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 13,
    "max_score": 1,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.4.0",
        "_score": 1,
        "_source": {
          "buildNum": 15063,
          "defaultIndex": "*",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.4.1",
        "_score": 1,
        "_source": {
          "buildNum": 15104,
          "defaultIndex": "*",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.4.2",
        "_score": 1,
        "_source": {
          "buildNum": 15117,
          "defaultIndex": "*",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.4.3",
        "_score": 1,
        "_source": {
          "buildNum": 15122,
          "defaultIndex": "logstash-*",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.5.0",
        "_score": 1,
        "_source": {
          "buildNum": 15382,
          "defaultIndex": "logstash-*",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.5.1",
        "_score": 1,
        "_source": {
          "buildNum": 15405,
          "defaultIndex": "logstash-*",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.5.2",
        "_score": 1,
        "_source": {
          "buildNum": 15443,
          "defaultIndex": "logstash-*",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.6.0",
        "_score": 1,
        "_source": {
          "buildNum": 15523,
          "defaultIndex": "logstash-*",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.6.1",
        "_score": 1,
        "_source": {
          "buildNum": 15533,
          "defaultIndex": "logstash-*",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.6.2",
        "_score": 1,
        "_source": {
          "buildNum": 15543,
          "defaultIndex": null,
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.6.3",
        "_score": 1,
        "_source": {
          "buildNum": 15554,
          "defaultIndex": "AV-QaIjQa6287cCjMl-I",
          "discover:aggs:terms:size": 20,
          "metaFields": null
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.6.4",
        "_score": 1,
        "_source": {
          "buildNum": 15571,
          "defaultIndex": "AWAgQOJuZcEx8N3pb4qE",
          "discover:aggs:terms:size": 20,
          "metaFields": null,
          "state:storeInSessionStorage": true
        }
      },
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.6.5",
        "_score": 1,
        "_source": {
          "buildNum": 15592,
          "defaultIndex": "AWAgQOJuZcEx8N3pb4qE",
          "discover:aggs:terms:size": 20,
          "metaFields": null,
          "state:storeInSessionStorage": true
        }
      }
    ]
  }
}

weltenwort · December 14, 2017, 10:02am

I can't see anything unusual in these settings. I'm sorry to say that without a way to reproduce it successfully, further debugging is very difficult. I couldn't find any similar reports of other users either. It would be interesting to see that result right after the phenomenon occurred before an index pattern was defined as the default index pattern again. Please update us when it occurs again.

rs_ela · December 14, 2017, 11:07am

Right. Will do.

Thank you for your help so far!

system · January 11, 2018, 11:07am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.