Kibana loading loop with .kibana-event-log has is_hidden set to true on indices

Hello everyone,

I will try to get some help on the forum before trying a last update or crash something more.
Here is my situation :
I had a cluster elastic with rpm on 7.10.1 and I upgraded nearly 2 months ago in 7.16.2
Everything was working fine during 2 months !!! but yesteardy my kibana was looping indefinitely when I open a new sessions.
I was getting theses errors on elastic nodes client where kibana connect.

[2022-02-22T10:22:00,721][WARN ][r.suppressed ] [ingest_host] path: /_aliases, params: {}
org.Elasticsearch.transport.RemoteTransportException: [master_host][ip:9300][indices:admin/aliases]
Caused by: java.lang.IllegalStateException: alias [.kibana-event-log-7.10.0] has is_hidden set to true on indices [.kibana-event-log-7.10.0-000014] but does not have is_hidden set to true on indices [.kibana-event-log-7
.10.0-000015,.kibana-event-log-7.10.0-000013,.kibana-event-log-7.10.0-000016]; alias must have the same is_hidden setting on all indices

after trying to update the field with no sucess

I deleted theses indices like theses post was saying

But in fact I didnt have .kibana-event-log-7.16.2 and when I deleted 7.10 it deleted every event-logs on the cluster !

So I'm trying to recreate theses indices from antoher cluster 7.16.2 with curl command but IT doesnt work.

chrome/firefox debug indicate no reponse on this request from Kibana interface
GET on https://kibana_host:5601/46307/bundles/plugin/runtimeFields/kibana/runtimeFields.plugin.js

and elastic node indicate no more errors!

[2022-02-23T16:04:11,573][INFO ][o.e.t.LoggingTaskListener] [ingest_host] 8870084 finished with response BulkByScrollResponse[took=79.2ms,timed_out=false,sliceId=null,updated=50,created=0,deleted=0,batches=1,versionConflicts=1,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-02-23T16:04:15,114][INFO ][o.e.t.LoggingTaskListener] [ingest_host] 8870092 finished with response BulkByScrollResponse[took=3.6s,timed_out=false,sliceId=null,updated=4264,created=0,deleted=0,batches=5,versionConflicts=3,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]

I verify account from kibana is ok.

I reinstall this template from fresh 7.16.2 install
.kibana-event-log-7.16.2-* from index template

I try to reisntall rpm 7.16.2 but it does nothing.
for the moment I have no snapshot of my kibana dashboard uptodate but some backup.
Can I had a snapshot repository by curl and backup now ?
Can I had reindex .kibana object or export them with another way. reisntall kibana 7.16.2 et reimport dashboard ?

is a update in 7.17 can fix this problem?

Thank you for your time and your answer in advance.
see ya
Kevin

I would say don't bother trying to re-create the .kibana-event-log* indices. They contain historic information about what the kibana alerting system has been doing, and Kibana can operate fine without the old ones.

Re-creating them by hand is likely difficult, as there is an ILM policy in play as well. In fact, if you try creating these indices, aliases, templates, etc, by hand, there's a chance you could break writing out the historic data in the new deployment.

Hello Patrick, thank you for taking time to respond to my problem. Yes .kibana-event-log indices were recreted but it doesn't fix my kibana problem. I think when I try to manipulate kibana system indices by curl, I made a wrong manipulation on .kibana_7.16.2_001 indices. I compare my indices with a fresh install on eck and I see few differences on mappings ! I m not sur this critical. I will try to put elastic et kibana and in debug to see which requests is looping. I will try to save my saved objects and delete kibana indices to reinstall kibana !
If you have a better idea ?

Good news kibana has been fully recover.
in order i try to add .kibana-event-log-7.16.2-template. not sure of the impact. but its a template it should not harm !
I put Elasticsearch in full debug.
And last but not least and it's true that I didnt reboot my windows laptop since 5 days since the crash. I had flush my data in the browser. close it and try another firefox instances but I didn't reboot.
so in last case try to reboot your laptop !!! ;D
keep the good work elastic community. see you next time.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.