Kibana index corrupted? Should I delete it and restart service?

Elastic Stack Kibana
1

A newly installed kibana could not create index pattern, so I restarted the service, and then I was not able to connect to 5601.

From the logs (see bellow) I suspect that there is a problem with the kibana indices. What if I deleted all ".kibana*" indices and restart the service - would it recreate it correctly?
Or perhaps a simple editing of a .kibana index settings would fix the issue?

{"type":"log","@timestamp":"2022-07-28T17:22:51+03:00","tags":["info","savedobjects-service"],"pid":1178361,"message":"[.kibana] INIT -> FATAL. took: 553ms."}
{"type":"log","@timestamp":"2022-07-28T17:22:51+03:00","tags":["fatal","root"],"pid":1178361,"message":"Error: Unable to complete saved object migrations for the [.kibana] index: The .kibana alias is pointing to a newer version of Kibana: v7.17.1\n    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:144:29)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 0)\n    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:181:9)\n    at Server.start (/usr/share/kibana/src/core/server/server.js:330:31)\n    at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)\n    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:229:5)"}

and here is what i get with curl -X GET localhost:9200/.kibana/_settings?pretty=true

{
  ".kibana_7.17.1_001" : {
    "settings" : {
      "index" : {
        "refresh_interval" : "1s",
        "auto_expand_replicas" : "0-1",
        "blocks" : {
          "write" : "false"
        },
        "provided_name" : ".kibana_7.17.1_001",
        "creation_date" : "1651065321600",
        "priority" : "10",
        "number_of_replicas" : "0",
        "uuid" : "i6VMkZwpRXyYGwTdES99QA",
        "version" : {
          "created" : "7170099"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            },
            "initial_recovery" : {
              "_id" : null
            }
          }
        },
        "number_of_shards" : "1",
        "routing_partition_size" : "1",
        "resize" : {
          "source" : {
            "name" : ".kibana_7.17.1_reindex_temp",
            "uuid" : "_Yxk0R1qRMuwbmcFybVUOw"
          }
        }
      }
    }
  }
}
2

Welcome to our community! :smiley:

Did you attempt any upgrades here?

3

Thanks warkolm!
I was not the one that did the installation and the person that did it is not available... So I can only guess that an upgrade started but never completed...

I also noticed two event logs from two versions both with "is_write_index" : true..

 ".kibana-event-log-7.17.0-000006" : {
    "aliases" : {
      ".kibana-event-log-7.17.0" : {
        "is_write_index" : true,
        "is_hidden" : true
      }
    }
  }

and

 ".kibana-event-log-7.17.1-000004" : {
    "aliases" : {
      ".kibana-event-log-7.17.1" : {
        "is_write_index" : true,
        "is_hidden" : true
      }
    }
  },
4

Do you have multiple Kibana instances connected to this cluster? Are they all running the same version? Given that the .kibana alias is already pointing to .kibana_7.17.1_001, running an older version of Kibana would throw the error you reported.

5

No, there is only one instance, "7.17.0"

In kibana logfile I can only find "kbn-version":"7.17.0".

If I search for "7.17.1" I only find reports of the index problem ("Error: Unable to complete saved object migrations for the [.kibana] index: The .kibana alias is pointing to a newer version of Kibana: 7.17.1\n"

So i am not sure how this index was created...

6

@Mark_S - Perhaps, it is the result of an upgrade attempt as @warkolm mentioned earlier. If only Kibana 7.17.0 is running in your environment, then you may want to "rollback" the Kibana aliases. In order to do that, you will need to perform a few steps:

  1. Stop Kibana

  2. Reset the index.blocks.write setting for the Kibana indices (it could have been set during the saved objects migration to 7.17.1):

PUT .kibana_7.17.0_001,.kibana_task_manager_7.17.0_001/_settings
{
  "index.blocks.write": null
}
  1. Change the aliases:
POST /_aliases
{
    "actions" : [
        { "remove" : { "index" : ".kibana_7.17.1_001", "alias" : ".kibana" } },
        { "add" : { "index" : ".kibana_7.17.0_001", "alias" : ".kibana" } }
    ]
}

(Optional - depends if you have the same problem for the .kibana_task_manager alias too)

POST /_aliases
{
    "actions" : [
        { "remove" : { "index" : ".kibana_task_manager_7.17.1_001", "alias" : ".kibana_task_manager" } },
        { "add" : { "index" : ".kibana_task_manager_7.17.0_001", "alias" : ".kibana_task_manager" } }
    ]
}
  1. Start Kibana and check that Kibana is up and running
1 Like
7

It worked like a charm!
Thank you for the detailed answer!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.