Kibana visualizations broken after deleting index pattern - modifying object does not work

[mujina93 (Michele Piccolini)]

Hello,

I had a dashboard in Kibana that was relying on an index pattern.

I set a lifetime policy on the index and my index got deleted (I thought the "deletion time" was a retention window, i.e. documents older than X days would be deleted. Instead I discovered with great horror that the meaning was "after X days, delete everything").

I created that again and now I have some new data. I recreated the index pattern in Kibana, but now all my visualizations are broken. (I get a blank screen when trying to edit or see the visualizations)

I already tried to edit the saved objects, adding the "index" field as suggested here: https://discuss.elastic.co/t/visualizations-broken-after-deleting-index-pattern-resolved-the-index-pattern-associated-with-this-object-no-longer-exists/181452 .

I Tried to put the new index pattern's uuid there, and also the index pattern's name itself (e.g. "something.*.something"), but none work.

I made sure that the fields used in the visualization object exist and are in the right format in the new index pattern.

Is there a way to debug/troubleshoot this? I would like to understand why the visualizations are not showing up. I have several of them and I cannot make all of them again from scratch.

Thank you in advance for your support.

[wylie (Wylie Conlon)]

The instructions you posted make sense to me: visualizations currently refer to the UUID of the index pattern saved object, so you need to update all your visualizations when the UUID has changed (such as when you create a new index pattern object).

There is another factor that might be affecting this, which is that index pattern names must be unique, because we lookup by name in some places and by ID in others. Visualizations do both, so if you have two duplicate names there could be an issue.

The best way to debug this is by trying to get just one visualization to work following the steps shown in that linked thread, and to validate that the index pattern ID is matched.

[aaron_nimocks]

Think what I would try next is to create a new visualization (blank). Has all the right config for the index.

Export object and now replace settings with old object data.

[mujina93 (Michele Piccolini)]

(I will bold my questions hereafter, so that they are easy to notice)

Thanks, what I did was indeed creating some new visualizations, exporting them, inspecting the ndjson, finding the reference field(s), copying the content and pasting inside the reference array (as suggested in the managing-saved-objects-in-kibana-7.7 guide) in the edit object view in Kibana. Much like what is suggested in change-visualization-index-under-saved-objects.

Modifying the index field under kibanaSavedObjectMeta.searchSourceJSON as suggested in visualizations-broken-after-deleting-index-pattern-resolved-the-index-pattern-associated-with-this-object-no-longer-exists is something which apparently is valid for kibana 6.8, judging by the part at the end in the guide managing-saved-objects-in-kibana-6.8.

However I still had to put my index's uuid in the reference, which visualizations-broken-after-deleting-index-pattern-resolved-the-index-pattern-associated-with-this-object-no-longer-exists suggests to take from the link to the index page under index management in kibana.

My problem was that I didn't have a simple single-object and easy-to-guess reference that could be solved just with a new index uuid. For example, I had a Control kind of visualization that required 3 entries in the reference array, each with a custom name. And I couldn't have guessed the exact required structure and naming just by looking at what is provided in the Edit Visualization view (Saved Objects > Inspect) alone, since when I edit objects (also valid and new ones!) from there, the reference entry is always empty.

I don't see why it should be like that, since when one then goes the more laborious way of downloading the export.ndarray, the reference can be found.

Why isn't all the information present in Kibana's view for editing objects?

Anyway, my question remains:

is there a way of debugging such issues?

Unfortunately, when a visualization is broken, it just doesn't show up (can't be visualized), and it does so silently. Even the guide managing-saved-objects-in-kibana-7.7 warns you at the end about the fact that even a little change might render the visualization unusable, and that manual editing should be avoided:

Validation is not performed for object properties. Submitting an invalid change will render the object unusable.

But what about the cases when I must edit manually? Is there a way to discover if it's a wrong type in one field, if it's wrong syntax, if it's a missing index's uuid, or wrong number of entries or format in the reference or in other fields?

And a final question related to this topic, which I ask in advance so that it won't bite me:

if I now introduce index aliases (I need rollover on my index), or change index name or index pattern name, or I change some field's meta-information (like index or type or keyword), what's the correct and cleanest way of propagating this change to my visualizations?
Can I do something in advance? (For example, I don't see any button or field to change the index pattern when I edit a visualization in the proper "UI" way)
Or will I have to go through this process of blind trial and error via manual editing of the saved object? (Now I know how to do it but it's a bit time consuming and error-prone).

Thanks for your support and thank you again for your quick answers!

[flash1293 (Joe Reuter)]

One way to change the index pattern for all visualizations at once is to note down the uuid of the old index pattern (you can see it in the reference array when inspecting a visualization), then delete the old index pattern and create the new one under the same id.

In Step 2 of the index pattern creation process you can specify the used id by expanding "advanced options":

If you do this make sure the new index pattern has the same config and fields as the deleted one, otherwise it's possible to run into problems with the visualization (e.g. if a visualization is referencing a field which is not available in the new index pattern).

[mujina93 (Michele Piccolini)]

Thanks @flash1293.

Just one comment: you say

@flash1293

the uuid of the old index pattern (you can see it in the reference array when inspecting a visualization)

but as I already said, I don't have anything under references, not even for new and perfectly functional visualizations. Here is what I see:

[flash1293 (Joe Reuter)]

If you haven’t deleted the old index pattern yet, you can find the uuid by selecting the index pattern in the index pattern management and check the url - the uuid will show up there as well.

In general: If your Saved object is too corrupted to be viewed in the “Inspect” view, you can also use the dev tools and Elasticsearch apis to directly work with the underlying document in the .kibana index. The document id is “visualization:”