Hi,
we recently migrated elk version from 7.0.1 to 7.8.0. In kibana 7.0.1, the document count for kibana index increases when we create any new saved object.
But with elk 7.8.0, we observed that the document count of kibana index increases even when there are no saved objects. On describing index, we got to know that with clicks and switch between tabs on kibana GUI, the document count kept increasing.
Can anyone from elastic confirm if this is an expected behaviour or a known issue?
Out of curiosity, can you provide the following?
GET _cat/indices/.kibana*GET _cat/aliases/.kibana*If you want to inspect these documents, you can search through them with GET .kibana/_search. We use these indices to persist data used in Kibana, so it's hard to say what exactly is resulting in the increased count. However, from my knowledge, we don't create documents when switching tabs. This index should not grow unbounded over time and normally remains pretty small. The largest .kibana index I have seen was in the 10's of thousands of documents. Normally I would expect something under 1k.
Hi @tylersmalley
Thanks for the response.
For the above query the output are :- 1
GET _cat/indices/.kibana*
green open .kibana_1 sWfnzlD_Sg-Xbimx0YTmpw 1 1 26 1 87.1kb 41.9kb
GET _cat/aliases/.kibana*
.kibana .kibana_1 - - - -
after some minutes i run _cat/indices/.kibana* query the count increases from 26 to 32 even the saved object is same
GET _cat/indices/.kibana*
green open .kibana_1 sWfnzlD_Sg-Xbimx0YTmpw 1 1 32 2 146.5kb 61.8kb
Even the count is 26 there are two saved objects.
Please refer the attachment for the saved objects.
That is odd. My guess is there is some background task manager job or something that is being kicked off. Seeing as your document count is only 32, it doesn't seem like something which is really a concern.
If you do want to know exactly what is occupying those documents you could capture a before and after of the documents in Elasticsearch and diff them. If you want to do that I could take a look.
GET .kibana/_search?size=100
Hi @tylersmalley
I am sharing some part of GET .kibana/_search?size=100 query. Could please look into it.
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:38511140-f1ab-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 2.3796833333333334,
"numberOfClicks" : 5,
"appId" : "discover",
"timestamp" : "2020-09-08T08:13:47.476Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-08T08:13:47.476Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:38511141-f1ab-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 0.43166666666666664,
"numberOfClicks" : 10,
"appId" : "management",
"timestamp" : "2020-09-08T08:13:47.476Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-08T08:13:47.476Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:0d6f33a0-f1a9-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 5.427433333333333,
"numberOfClicks" : 10,
"appId" : "discover",
"timestamp" : "2020-09-08T07:58:16.538Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-08T07:58:16.538Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:7237e9b0-f1ab-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 1.6461666666666666,
"numberOfClicks" : 13,
"appId" : "discover",
"timestamp" : "2020-09-08T08:15:24.619Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-08T08:15:24.619Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:099b9990-f1ad-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 6.076033333333333,
"numberOfClicks" : 0,
"appId" : "discover",
"timestamp" : "2020-09-08T08:26:48.105Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-08T08:26:48.105Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:823b9960-f1b5-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 53.5083,
"numberOfClicks" : 2,
"appId" : "discover",
"timestamp" : "2020-09-08T09:27:26.454Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-08T09:27:26.454Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:fa32f281-f280-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 0.22506666666666666,
"numberOfClicks" : 2,
"appId" : "home",
"timestamp" : "2020-09-09T09:43:55.560Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-09T09:43:55.560Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:fa32f282-f280-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 0.07123333333333333,
"numberOfClicks" : 2,
"appId" : "dashboards",
"timestamp" : "2020-09-09T09:43:55.560Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-09T09:43:55.560Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:fa32f283-f280-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 0.15801666666666667,
"numberOfClicks" : 4,
"appId" : "discover",
"timestamp" : "2020-09-09T09:43:55.560Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-09T09:43:55.560Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:fa32f284-f280-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 0.09258333333333334,
"numberOfClicks" : 2,
"appId" : "visualize",
"timestamp" : "2020-09-09T09:43:55.560Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-09T09:43:55.560Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:3acf2b60-f281-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 0.060583333333333336,
"numberOfClicks" : 2,
"appId" : "management",
"timestamp" : "2020-09-09T09:45:43.957Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-09T09:45:43.958Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "application_usage_transactional:3acf2b61-f281-11ea-91df-51d652450a7f",
"_score" : 1.0,
"_source" : {
"application_usage_transactional" : {
"minutesOnScreen" : 0.2377,
"numberOfClicks" : 3,
"appId" : "dev_tools",
"timestamp" : "2020-09-09T09:45:43.957Z"
},
"type" : "application_usage_transactional",
"references" : [ ],
"updated_at" : "2020-09-09T09:45:43.958Z"
}
},
{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "ui-metric:console:GET_cat.aliases",
"_score" : 1.0,
"_source" : {
"ui-metric" : {
"count" : 1
},
"type" : "ui-metric",
"updated_at" : "2020-09-09T09:47:31.847Z"
}
},It looks like this data is for usage collection which helps us understand how to improve the product. It doesn't appear you can disable the creation of these documents, and it shouldn't grow unbounded.
If you would like to disable the sending of telemetry data, you can do so by setting telemetry.enabled in the kibana.yml. More details here.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.