# Kibana loading loop with .kibana-event-log has is\_hidden set to true on indices

**URL:** https://discuss.elastic.co/t/kibana-loading-loop-with-kibana-event-log-has-is-hidden-set-to-true-on-indices/298088
**Category:** Kibana
**Created:** [February 23, 2022, 9:56pm UTC](https://discuss.elastic.co/t/kibana-loading-loop-with-kibana-event-log-has-is-hidden-set-to-true-on-indices/298088 "2022-02-23T21:56:58Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![kevb](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@kevb](https://discuss.elastic.co/u/kevb)
#### Post date: [February 23, 2022, 9:56pm UTC](https://discuss.elastic.co/t/kibana-loading-loop-with-kibana-event-log-has-is-hidden-set-to-true-on-indices/298088/1 "2022-02-23T21:56:58Z")

</div>

Hello everyone,

I will try to get some help on the forum before trying a last update or crash something more.  
Here is my situation :  
I had a cluster elastic with rpm on 7.10.1 and I upgraded nearly 2 months ago in 7.16.2  
Everything was working fine during 2 months !!! but yesteardy my kibana was looping indefinitely when I open a new sessions.  
I was getting theses errors on elastic nodes client where kibana connect.

[2022-02-22T10:22:00,721][WARN][r.suppressed] [ingest\_host] path: /\_aliases, params: {}  
org.elasticsearch.transport.RemoteTransportException: [master\_host][ip:9300][indices:admin/aliases]  
Caused by: java.lang.IllegalStateException: alias [.kibana-event-log-7.10.0] has is\_hidden set to true on indices [.kibana-event-log-7.10.0-000014] but does not have is\_hidden set to true on indices [.kibana-event-log-7  
.10.0-000015,.kibana-event-log-7.10.0-000013,.kibana-event-log-7.10.0-000016]; alias must have the same is\_hidden setting on all indices

after trying to update the field with no sucess

> <https://github.com/elastic/kibana/pull/122882>
>
> Resolves https://github.com/elastic/kibana/issues/121298
> 
> \## Summary
> 
> When u…pgrading from \<= 7.15 versions of Kibana, the event log will try to update existing event log indices to \`hidden\`. Part of this process involves setting existing index aliases to \`is\_hidden\`. It turns out if you have one alias pointing to multiple indices, we need to update the alias settings in a single command instead of consecutive commands as we were doing previously.
> 
> For example, if our 7.15 event log aliases look like this:
> 
> \`\`\`
> {
> ".kibana-event-log-7.15.2-000003": {
> "aliases": {
> ".kibana-event-log-7.15.2": {
> "is\_write\_index": true
> }
> }
> },
> ".kibana-event-log-7.15.2-000002": {
> "aliases": {
> ".kibana-event-log-7.15.2": {
> "is\_write\_index": false
> }
> }
> },
> ".kibana-event-log-7.15.2-000001": {
> "aliases": {
> ".kibana-event-log-7.15.2": {
> "is\_write\_index": false
> }
> }
> }
> }
> \`\`\`
> 
> Previously, we were issuing 3 calls to the update alias API, each one looking like this: 
> 
> \`\`\`
> {
> actions: \[{
> add: {
> index: \<indexName\>,
> alias: '.kibana-event-log-7.15.2',
> is\_hidden: true,
> },
> }, \],
> }
> \`\`\`
> 
> We actually need to be issuing 1 call to the update alias API that looks like this:
> 
> \`\`\`
> {
> "actions": \[{
> "add": {
> "is\_write\_index": false,
> "index": ".kibana-event-log-7.15.2-000002",
> "alias": ".kibana-event-log-7.15.2",
> "is\_hidden": true
> }
> },
> {
> "add": {
> "is\_write\_index": true,
> "index": ".kibana-event-log-7.15.2-000003",
> "alias": ".kibana-event-log-7.15.2",
> "is\_hidden": true
> }
> },
> {
> "add": {
> "is\_write\_index": false,
> "index": ".kibana-event-log-7.15.2-000001",
> "alias": ".kibana-event-log-7.15.2",
> "is\_hidden": true
> }
> }
> \]
> }
> \`\`\`
> 
> This PR updates to do this correctly.
> 
> \## To Verify
> 
> 1. Change the Event log ILM policy to max\_docs: 1 (to get ILM to roll over faster). Run 7.15 (or previous version) and let some rules/actions run to generate some event log docs and get multiple event log indices.
> 2. Run \`main\` using the previous data and notice that there is an error in the logs when updating the index aliases that cause the aliases not to get updated to \`hidden\`.
> 3. Run this branch using the previous data and notice no errors in the logs that verify that the index aliases are all updated to hidden.
> 
> \### Checklist
> 
> \- \[x\] \[Unit or functional tests\](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

I deleted theses indices like theses post was saying

> [@Deleting Index kibana](https://discuss.elastic.co/t/deleting-index-kibana/293773):
>
> Hello there, Following log entry is appearing in kibana logs. I understand this is because of previous kibana versions. can I delete these indexes? Will deleting this affect my cluster? log [15:17:55.850] [error][eventLog][plugins] error setting existing ".kibana-event-log-7.8.1-000010" index aliases - error setting existing index aliases for index .kibana-event-log-7.8.1-000010 to is\_hidden: illegal\_state\_exception: [illegal\_state\_exception] Reason: alias [.kibana-event-log-7.8.1] has is\_hi…

But in fact I didnt have .kibana-event-log-7.16.2 and when I deleted 7.10 it deleted every event-logs on the cluster !

So I'm trying to recreate theses indices from antoher cluster 7.16.2 with curl command but IT doesnt work.

chrome/firefox debug indicate no reponse on this request from Kibana interface  
GET on https://kibana\_host:5601/46307/bundles/plugin/runtimeFields/kibana/runtimeFields.plugin.js

and elastic node indicate no more errors!

```auto
[2022-02-23T16:04:11,573][INFO][o.e.t.LoggingTaskListener] [ingest_host] 8870084 finished with response BulkByScrollResponse[took=79.2ms,timed_out=false,sliceId=null,updated=50,created=0,deleted=0,batches=1,versionConflicts=1,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-02-23T16:04:15,114][INFO][o.e.t.LoggingTaskListener] [ingest_host] 8870092 finished with response BulkByScrollResponse[took=3.6s,timed_out=false,sliceId=null,updated=4264,created=0,deleted=0,batches=5,versionConflicts=3,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]

```

I verify account from kibana is ok.

I reinstall this template from fresh 7.16.2 install  
.kibana-event-log-7.16.2-\* from index template

I try to reisntall rpm 7.16.2 but it does nothing.  
for the moment I have no snapshot of my kibana dashboard uptodate but some backup.  
Can I had a snapshot repository by curl and backup now ?  
Can I had reindex .kibana object or export them with another way. reisntall kibana 7.16.2 et reimport dashboard ?

is a update in 7.17 can fix this problem?

Thank you for your time and your answer in advance.  
see ya  
Kevin

---

<div class="post-metadata">

### Author: ![Patrick\_Mueller](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick_mueller/32/109425_2.png) [@Patrick\_Mueller](https://discuss.elastic.co/u/Patrick_Mueller)
#### Post date: [February 24, 2022, 3:13pm UTC](https://discuss.elastic.co/t/kibana-loading-loop-with-kibana-event-log-has-is-hidden-set-to-true-on-indices/298088/2 "2022-02-24T15:13:32Z")

</div>

I would say don't bother trying to re-create the `.kibana-event-log*` indices. They contain historic information about what the kibana alerting system has been doing, and Kibana can operate fine without the old ones.

Re-creating them by hand is likely difficult, as there is an ILM policy in play as well. In fact, if you try creating these indices, aliases, templates, etc, by hand, there's a chance you could break writing out the historic data in the new deployment.

---

<div class="post-metadata">

### Author: ![kevb](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@kevb](https://discuss.elastic.co/u/kevb)
#### Post date: [February 24, 2022, 8:14pm UTC](https://discuss.elastic.co/t/kibana-loading-loop-with-kibana-event-log-has-is-hidden-set-to-true-on-indices/298088/3 "2022-02-24T20:14:25Z")

</div>

Hello Patrick, thank you for taking time to respond to my problem. Yes .kibana-event-log indices were recreted but it doesn't fix my kibana problem. I think when I try to manipulate kibana system indices by curl, I made a wrong manipulation on .kibana\_7.16.2\_001 indices. I compare my indices with a fresh install on eck and I see few differences on mappings ! I m not sur this critical. I will try to put elastic et kibana and in debug to see which requests is looping. I will try to save my saved objects and delete kibana indices to reinstall kibana !  
If you have a better idea ?

---

<div class="post-metadata">

### Author: ![kevb](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@kevb](https://discuss.elastic.co/u/kevb)
#### Post date: [February 24, 2022, 10:00pm UTC](https://discuss.elastic.co/t/kibana-loading-loop-with-kibana-event-log-has-is-hidden-set-to-true-on-indices/298088/4 "2022-02-24T22:00:20Z")

</div>

Good news kibana has been fully recover.  
in order i try to add .kibana-event-log-7.16.2-template. not sure of the impact. but its a template it should not harm !  
I put Elasticsearch in full debug.  
And last but not least and it's true that I didnt reboot my windows laptop since 5 days since the crash. I had flush my data in the browser. close it and try another firefox instances but I didn't reboot.  
so in last case try to reboot your laptop !!! ;D  
keep the good work elastic community. see you next time.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 24, 2022, 10:00pm UTC](https://discuss.elastic.co/t/kibana-loading-loop-with-kibana-event-log-has-is-hidden-set-to-true-on-indices/298088/5 "2022-03-24T22:00:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.