Changing the index pattern that a visualisation is based on

[Russell_Fulton]

I have found a set of visualisation on github that I want to use. But they are build around a different pattern of index name to what I have even though the mapping are the same (or at least mostly).

I have the ndjson files for the dashboards and can see find the index patterns in there. Can I replace these and then reload the templates with the overwrite option?

Are there any plans make the index patterns configurable? It would make the sharing of templates much easier.

[flash1293 (Joe Reuter)]

Editing the ndjson file is certainly an option. Each visualization has a "references" section - if you already have the index pattern saved object in your Kibana instance, you can simply fill in your id there:

"references":[{"id":"90943e30-9a47-11e8-b64d-95841ca0b247","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}]

replace by

"references":[{"id":"the-id-of-your-local-index-pattern","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}]

I also explained some approaches here which work from within the Kibana UI: Rename the index-pattern for sample visualizations - #2 by flash1293

Lens visualizations allow you to change the index pattern simply by editing them and changing it in the dropdown.

[Russell_Fulton]

Thanks Joe!

BTW I am using ES/Kibana 7.10 if that matters... I also note that the "id" I have in the json is the actual name of the index pattern, not a "Unique ref" as in your examples.

  {
    "id": "logstash-alert-*",
    "name": "kibanaSavedObjectMeta.searchSourceJSON.index",
    "type": "index-pattern"

}

I first tried editing the ndjson file to change the "id" field to my target index pattern which is "evebox". evebox is an alias that finds all the relevant files and works fine in discover. This is also what was in the url when I looked at "Stack management > index patterns > "

After a lot of stuffing around I found the proper ID for the evebox pattern by looking at "Stack management > saved objects > evebox" and clicking inspect.

Armed with that I did a global change on the ndjson file and it all burst into life.

I suspect this is a difference in ES versions!

I did try recreating the index pattern but since I did not have any indexes matching that pattern I could not create the index in kibana. I suspect I could have added the necessary fields to the json that was used to create the index patterns in the first place. (these came with the dashboard set).

It may be worthwhile to update your excellent blog with information about Version 7 (if that was the issue)

Thanks again

[flash1293 (Joe Reuter)]

Index patterns in Kibana are confusing on many levels, but you figured it out! I'll try to sum it up.

First, you have to know about "saved objects" - these are pieces of meta information specific to Kibana. They are also stored in Elasticsearch, but they are not an Elasticsearch functionality (they are just regular documents). A visualization is a saved object (storing information like which chart type to use and how to fetch data), a dashboard is a saved object (storing which visualizations to use and the position of their panels), and - that's the weird part - an "index pattern" is a saved object, too.

It stores information like how to format which field and also the actual "pattern" (like logstash-alert-*) which is used to find the data indices in Elasticsearch. Each saved object has an id itself (it's just the id of the document where it's stored). In older versions of Kibana, the index pattern saved objects used the pattern itself as id (because it's probably unique already, so why not). This caused a bunch of issues, so more recent versions of Kibana don't do that anymore and use an auto-generated id for the index pattern saved object (the pattern itself is stored in the "title" field).

I did try recreating the index pattern but since I did not have any indexes matching that pattern I could not create the index in kibana

True, Kibana requires you to have at least one index matching your pattern to create the saved object. You can always import index pattern saved objects that don't match though.

I could have added the necessary fields to the json that was used to create the index patterns in the first place. (these came with the dashboard set).

In older versions of Kibana that's the case, because the index pattern saved object has another functionality I skipped - it also caches the list of fields from the mappings of the data indices. However, messing with this list manually is brittle because someone could always "refresh" the field list from the UI. In more recent versions of Kibana (starting from 7.11), index patterns don't cache the field list anymore, but always pull this information from the source of truth - the mappings of the data indices.

[Russell_Fulton]

thanks for the detailed explanation! Post like this are invaluable because (as I am sure you are aware ; ) there is very little detailed overview anywhere in the manuals (ES is not alone here!)

I seem to spend my life trying to work out how to do stuff with APIs with hundreds of calls each of which has a tiny part of the story! Like trying to put a pages of a book in order without page numbers!

Thanks again!

[Russell_Fulton]

I discovered a trivially easy way to change the change the index pattern associated with a saved object where the original is invalid.

You simply export and then immediately import it again and you get this dialogue:

which allows you to select a replacement from your existing index patterns. I discovered this when I had one of my index patterns get corrupted somehow and had to delete and replace it leaving all my dashboards broken. I found the id of the new index pattern from the url as described in your instructions exported the dashboards, did a global search and replace and re-imported the file.

I managed to miss one instance with the replace and got the prompt.

for the next dashboard I tried export then import and it worked fine.

[flash1293 (Joe Reuter)]

That's a great approach, thanks for sharing. Note that it will only work if the referenced index pattern does not exist in the cluster, otherwise it won't show the dialog and simply accept the existing reference.