Kibana loading loop with .kibana-event-log has is_hidden set to true on indices

kevb · February 23, 2022, 9:56pm

Hello everyone,

I will try to get some help on the forum before trying a last update or crash something more.
Here is my situation :
I had a cluster elastic with rpm on 7.10.1 and I upgraded nearly 2 months ago in 7.16.2
Everything was working fine during 2 months !!! but yesteardy my kibana was looping indefinitely when I open a new sessions.
I was getting theses errors on elastic nodes client where kibana connect.

[2022-02-22T10:22:00,721][WARN ][r.suppressed ] [ingest_host] path: /_aliases, params: {}
org.elasticsearch.transport.RemoteTransportException: [master_host][ip:9300][indices:admin/aliases]
Caused by: java.lang.IllegalStateException: alias [.kibana-event-log-7.10.0] has is_hidden set to true on indices [.kibana-event-log-7.10.0-000014] but does not have is_hidden set to true on indices [.kibana-event-log-7
.10.0-000015,.kibana-event-log-7.10.0-000013,.kibana-event-log-7.10.0-000016]; alias must have the same is_hidden setting on all indices

after trying to update the field with no sucess

github.com/elastic/kibana

[Event Log] Fixing call to update index alias to hidden

elastic:main ← ymao1:event-log/hidden-again

opened 08:01PM - 12 Jan 22 UTC

ymao1

+142 -62

Resolves https://github.com/elastic/kibana/issues/121298 ## Summary When u…pgrading from <= 7.15 versions of Kibana, the event log will try to update existing event log indices to `hidden`. Part of this process involves setting existing index aliases to `is_hidden`. It turns out if you have one alias pointing to multiple indices, we need to update the alias settings in a single command instead of consecutive commands as we were doing previously. For example, if our 7.15 event log aliases look like this: ``` { ".kibana-event-log-7.15.2-000003": { "aliases": { ".kibana-event-log-7.15.2": { "is_write_index": true } } }, ".kibana-event-log-7.15.2-000002": { "aliases": { ".kibana-event-log-7.15.2": { "is_write_index": false } } }, ".kibana-event-log-7.15.2-000001": { "aliases": { ".kibana-event-log-7.15.2": { "is_write_index": false } } } } ``` Previously, we were issuing 3 calls to the update alias API, each one looking like this: ``` { actions: [{ add: { index: <indexName>, alias: '.kibana-event-log-7.15.2', is_hidden: true, }, }, ], } ``` We actually need to be issuing 1 call to the update alias API that looks like this: ``` { "actions": [{ "add": { "is_write_index": false, "index": ".kibana-event-log-7.15.2-000002", "alias": ".kibana-event-log-7.15.2", "is_hidden": true } }, { "add": { "is_write_index": true, "index": ".kibana-event-log-7.15.2-000003", "alias": ".kibana-event-log-7.15.2", "is_hidden": true } }, { "add": { "is_write_index": false, "index": ".kibana-event-log-7.15.2-000001", "alias": ".kibana-event-log-7.15.2", "is_hidden": true } } ] } ``` This PR updates to do this correctly. ## To Verify 1. Change the Event log ILM policy to max_docs: 1 (to get ILM to roll over faster). Run 7.15 (or previous version) and let some rules/actions run to generate some event log docs and get multiple event log indices. 2. Run `main` using the previous data and notice that there is an error in the logs when updating the index aliases that cause the aliases not to get updated to `hidden`. 3. Run this branch using the previous data and notice no errors in the logs that verify that the index aliases are all updated to hidden. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

I deleted theses indices like theses post was saying

But in fact I didnt have .kibana-event-log-7.16.2 and when I deleted 7.10 it deleted every event-logs on the cluster !

So I'm trying to recreate theses indices from antoher cluster 7.16.2 with curl command but IT doesnt work.

chrome/firefox debug indicate no reponse on this request from Kibana interface
GET on https://kibana_host:5601/46307/bundles/plugin/runtimeFields/kibana/runtimeFields.plugin.js

and elastic node indicate no more errors!

[2022-02-23T16:04:11,573][INFO ][o.e.t.LoggingTaskListener] [ingest_host] 8870084 finished with response BulkByScrollResponse[took=79.2ms,timed_out=false,sliceId=null,updated=50,created=0,deleted=0,batches=1,versionConflicts=1,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-02-23T16:04:15,114][INFO ][o.e.t.LoggingTaskListener] [ingest_host] 8870092 finished with response BulkByScrollResponse[took=3.6s,timed_out=false,sliceId=null,updated=4264,created=0,deleted=0,batches=5,versionConflicts=3,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]

I verify account from kibana is ok.

I reinstall this template from fresh 7.16.2 install
.kibana-event-log-7.16.2-* from index template

I try to reisntall rpm 7.16.2 but it does nothing.
for the moment I have no snapshot of my kibana dashboard uptodate but some backup.
Can I had a snapshot repository by curl and backup now ?
Can I had reindex .kibana object or export them with another way. reisntall kibana 7.16.2 et reimport dashboard ?

is a update in 7.17 can fix this problem?

Thank you for your time and your answer in advance.
see ya
Kevin

Patrick_Mueller · February 24, 2022, 3:13pm

I would say don't bother trying to re-create the .kibana-event-log* indices. They contain historic information about what the kibana alerting system has been doing, and Kibana can operate fine without the old ones.

Re-creating them by hand is likely difficult, as there is an ILM policy in play as well. In fact, if you try creating these indices, aliases, templates, etc, by hand, there's a chance you could break writing out the historic data in the new deployment.

kevb · February 24, 2022, 8:14pm

Hello Patrick, thank you for taking time to respond to my problem. Yes .kibana-event-log indices were recreted but it doesn't fix my kibana problem. I think when I try to manipulate kibana system indices by curl, I made a wrong manipulation on .kibana_7.16.2_001 indices. I compare my indices with a fresh install on eck and I see few differences on mappings ! I m not sur this critical. I will try to put elastic et kibana and in debug to see which requests is looping. I will try to save my saved objects and delete kibana indices to reinstall kibana !
If you have a better idea ?

kevb · February 24, 2022, 10:00pm

Good news kibana has been fully recover.
in order i try to add .kibana-event-log-7.16.2-template. not sure of the impact. but its a template it should not harm !
I put Elasticsearch in full debug.
And last but not least and it's true that I didnt reboot my windows laptop since 5 days since the crash. I had flush my data in the browser. close it and try another firefox instances but I didn't reboot.
so in last case try to reboot your laptop !!! ;D
keep the good work elastic community. see you next time.

system · March 24, 2022, 10:00pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.