Importing dashboard and index pattern id

Hi,

I have an issue related to importing Kibana dashboard. My dashboard contains visualizations from different indices. When I import & export the dashboard using UI everything works fine. But when I export and then import the dashboard using API the visualizations are not displayed and they prompted the following error
Could not locate that index-pattern (id: .....

I also noticed that when I import the dashboard using the API with /import?exclude=index-pattern as in
#curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @$kibana_dashboard ${kibanaurl}/api/kibana/dashboards/import?exclude=index-pattern
The visualization works fine but replications of the index patterns are generated. I guess these auto-generated index patterns correspond to the old indices in the kibana dashboard.

I wonder if there is a way to make the import works without auto-generate the index patterns.

To add more details.
I am using elasticsearch 7.3.2 and Kibana 7.3.2
My pipeline as follows:

  • Cleaning any previous elastic indices
  • index all my file using elastic api _bulk
  • register all the indices as kibana index patterns
  • import the kibana dashboard

Note: I am using the same title for indices that kibana dashboard has.

I want an automatic way to recreate the index patterns that the visualizations in the imported dashboard are using and are missing.

Thanks

The problem here is visualizations in dashboards are referencing index patterns via auto-generated id instead of name - if you recreate them manually in Kibana they will get different ids than the references in the exported dashboard, so the dashboard won't render.

If you export via UI, the index pattern saved object will be part of the export because all referenced objects are included automatically. You can use the same mechanism via API as well by using the new saved objects api: https://www.elastic.co/guide/en/kibana/7.3/saved-objects-api.html

Thank you Joe for your reply.
If I understood you correctly, the way of importing or exporting the dashboard via (UI or API) matters in mapping the indices in the index pattern and the indices in the dashboard visualizations.

I actually created all ES indices and index patterns using API. I then created the dashboard and export it via API.

In reusing the dashboard, I tried to test this by firstly cleaning all indices, then created ES indices and index patterns using API, and importing the dashboard via API. I can see that the visualization works, however, I can also see one of the indices (index pattern name) in management > index patterns are repeated. So now, I have duplicated index patterns, how can I check the reason behind this?

I had same thing happen for few index pattern/dashboard. since then what I have been doing is

create full dashboard. export whole thing (which will include index pattern as well) and then import that in to different space or in different cluster.

if different cluster needs different name I generally just edit names (title etc..) and then import.

this way index pattern created there will have same id.

if I have complete different name/id/pattern but same dashboard then I do it individually.

for example five visulization using same type of index patter gets imported, then two more with different index pattern and then dashboard.

In reusing the dashboard, I tried to test this by firstly cleaning all indices, then created ES indices and index patterns using API, and importing the dashboard via API

If you import the dashboard via saved objects API with included referenced objects, the index patterns will be created for you and you don't have to create them manually. Just import via API and you are set.

I was using export/ import dashboard with kibana_url/api/kibana/dashboards/import, I will check on exporting dashboard from saved objects.

This is the legacy api which is not including all referenced objects. If possible, it shouldn't be used anymore.

Hello again @flash1293,

I tried to use the saved objects API for import and export.
My import request is

curl -X POST ${kibana_url}/saved_objects/_export -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '
{
  "objects": [
    {
      "type": "dashboard",
      "id": "123...."
    }
  ],
  "includeReferencesDeep": true
}'

But this throughs an error {"statusCode":404,"error":"Not Found","message":"Not Found"}%

My settings are that my old visualization was imported via the old legacy API, then I upload it and amended it then start using the saved object requests on the new dashboard.

Do saved objects call follows the same principles as importing/exporting via UI and kibana dashboard API, such that the dashboard has to be created from scratch and using desired import/export call to get it to work.

I think you are missing a /api in front of the save_objects here.

Do saved objects call follows the same principles as importing/exporting via UI and kibana dashboard API, such that the dashboard has to be created from scratch and using desired import/export call to get it to work.

Both the visualizations and the dashboard using them can be uploaded in the same request using the bulk request update API: https://www.elastic.co/guide/en/kibana/master/saved-objects-api-bulk-create.html

Thank you so much @flash1293
I have a further question. I want to reuse the same dashboard with a different index to the one I used to generate it. The new index will have the same data model but a different index name, and I want to end up with two identical dashboards that each represent a different index.

I want to achieve this by following this pipeline: indexing to ES, registering index-patterns, and then importing the dashboard. As the exported dashboard already have all the information for the index pattern (title and autogenerated ID) and the visualization (autogenerated ID) as in the code below, to make the second dashboard working with the new index, I need to recreate the title and the ID to match the index patterns of (the second ES index).

    "attributes":
    {
        "fields": "xxx",
        "timeFieldName": "",
        "title": "shops_index"
    },
    "id": "3f3ee7d0-29c7-11eb-9bb9-73a5cc088e4c",
    "migrationVersion":
    {
        "index-pattern": "6.5.0"
    },
    "references": [],
    "type": "index-pattern",
    "updated_at": "2020-11-25T15:09:25.722Z",
    "version": "WzI1LDFd"
}
{
    "attributes":
    {
        "description": "",
        "kibanaSavedObjectMeta":
        {
			...
        "title": "Graph 1",
        ...
       },
    "id": "2b04e070-29c8-11eb-9bb9-73a5cc088e4c",
    "migrationVersion":
    {
        "visualization": "7.3.1"
    },
    "references": [
    {
        "id": "3f3ee7d0-29c7-11eb-9bb9-73a5cc088e4c",
        "name": "kibanaSavedObjectMeta.searchSourceJSON.index",
        "type": "index-pattern"
    }],
    "type": "visualization",
    "updated_at": "2020-11-25T15:09:25.722Z",
    "version": "WzIwLDFd"
}```


To get the auto-generated ID of the index pattern I am using

GET .kibana/_search
{
  "_source": ["index-pattern.title"],
  "query": {
    "term": {
      "type": "index-pattern"
    }
  }
}

Then I am using these IDs to update the dashboard so the visualizations reflect the new indices.

I wonder if this the right way to reuse the dashboard?
I inspected the dashboard attributes in the saved object, I only updated the dashboard name and the ID of referenced visualizations, but it did not work, are there other configurations I need to update for dashboard reuse.

Thanks

Hi again @flash1293, can you assist with this. Thanks.

Hey, sorry for the late reply.

The approach you describe sounds right to me - for each of the visualizations, you need to change the id of the referenced index pattern, for the dashboard you need to change the ids of the referenced visualizations.

What exactly is not working?