Importing dashboard and index pattern id

Elastic Stack Kibana
1

Hi,

I have an issue related to importing Kibana dashboard. My dashboard contains visualizations from different indices. When I import & export the dashboard using UI everything works fine. But when I export and then import the dashboard using API the visualizations are not displayed and they prompted the following error
Could not locate that index-pattern (id: .....

I also noticed that when I import the dashboard using the API with /import?exclude=index-pattern as in
#curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @$kibana_dashboard ${kibanaurl}/api/kibana/dashboards/import?exclude=index-pattern
The visualization works fine but replications of the index patterns are generated. I guess these auto-generated index patterns correspond to the old indices in the kibana dashboard.

I wonder if there is a way to make the import works without auto-generate the index patterns.

To add more details.
I am using elasticsearch 7.3.2 and Kibana 7.3.2
My pipeline as follows:

  • Cleaning any previous elastic indices
  • index all my file using elastic api _bulk
  • register all the indices as kibana index patterns
  • import the kibana dashboard

Note: I am using the same title for indices that kibana dashboard has.

I want an automatic way to recreate the index patterns that the visualizations in the imported dashboard are using and are missing.

Thanks

2

The problem here is visualizations in dashboards are referencing index patterns via auto-generated id instead of name - if you recreate them manually in Kibana they will get different ids than the references in the exported dashboard, so the dashboard won't render.

If you export via UI, the index pattern saved object will be part of the export because all referenced objects are included automatically. You can use the same mechanism via API as well by using the new saved objects api: https://www.elastic.co/guide/en/kibana/7.3/saved-objects-api.html

3

Thank you Joe for your reply.
If I understood you correctly, the way of importing or exporting the dashboard via (UI or API) matters in mapping the indices in the index pattern and the indices in the dashboard visualizations.

I actually created all ES indices and index patterns using API. I then created the dashboard and export it via API.

In reusing the dashboard, I tried to test this by firstly cleaning all indices, then created ES indices and index patterns using API, and importing the dashboard via API. I can see that the visualization works, however, I can also see one of the indices (index pattern name) in management > index patterns are repeated. So now, I have duplicated index patterns, how can I check the reason behind this?

4

I had same thing happen for few index pattern/dashboard. since then what I have been doing is

create full dashboard. export whole thing (which will include index pattern as well) and then import that in to different space or in different cluster.

if different cluster needs different name I generally just edit names (title etc..) and then import.

this way index pattern created there will have same id.

if I have complete different name/id/pattern but same dashboard then I do it individually.

for example five visulization using same type of index patter gets imported, then two more with different index pattern and then dashboard.

5

In reusing the dashboard, I tried to test this by firstly cleaning all indices, then created ES indices and index patterns using API, and importing the dashboard via API

If you import the dashboard via saved objects API with included referenced objects, the index patterns will be created for you and you don't have to create them manually. Just import via API and you are set.

6

I was using export/ import dashboard with kibana_url/api/kibana/dashboards/import, I will check on exporting dashboard from saved objects.

7

This is the legacy api which is not including all referenced objects. If possible, it shouldn't be used anymore.

8

Hello again @flash1293,

I tried to use the saved objects API for import and export.
My import request is

curl -X POST ${kibana_url}/saved_objects/_export -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '
{
  "objects": [
    {
      "type": "dashboard",
      "id": "123...."
    }
  ],
  "includeReferencesDeep": true
}'

But this throughs an error {"statusCode":404,"error":"Not Found","message":"Not Found"}%

My settings are that my old visualization was imported via the old legacy API, then I upload it and amended it then start using the saved object requests on the new dashboard.

Do saved objects call follows the same principles as importing/exporting via UI and kibana dashboard API, such that the dashboard has to be created from scratch and using desired import/export call to get it to work.

9

I think you are missing a /api in front of the save_objects here.

Do saved objects call follows the same principles as importing/exporting via UI and kibana dashboard API, such that the dashboard has to be created from scratch and using desired import/export call to get it to work.

Both the visualizations and the dashboard using them can be uploaded in the same request using the bulk request update API: Bulk create saved objects API | Kibana Guide [master] | Elastic

10

Thank you so much @flash1293
I have a further question. I want to reuse the same dashboard with a different index to the one I used to generate it. The new index will have the same data model but a different index name, and I want to end up with two identical dashboards that each represent a different index.

I want to achieve this by following this pipeline: indexing to ES, registering index-patterns, and then importing the dashboard. As the exported dashboard already have all the information for the index pattern (title and autogenerated ID) and the visualization (autogenerated ID) as in the code below, to make the second dashboard working with the new index, I need to recreate the title and the ID to match the index patterns of (the second ES index).

    "attributes":
    {
        "fields": "xxx",
        "timeFieldName": "",
        "title": "shops_index"
    },
    "id": "3f3ee7d0-29c7-11eb-9bb9-73a5cc088e4c",
    "migrationVersion":
    {
        "index-pattern": "6.5.0"
    },
    "references": [],
    "type": "index-pattern",
    "updated_at": "2020-11-25T15:09:25.722Z",
    "version": "WzI1LDFd"
}
{
    "attributes":
    {
        "description": "",
        "kibanaSavedObjectMeta":
        {
			...
        "title": "Graph 1",
        ...
       },
    "id": "2b04e070-29c8-11eb-9bb9-73a5cc088e4c",
    "migrationVersion":
    {
        "visualization": "7.3.1"
    },
    "references": [
    {
        "id": "3f3ee7d0-29c7-11eb-9bb9-73a5cc088e4c",
        "name": "kibanaSavedObjectMeta.searchSourceJSON.index",
        "type": "index-pattern"
    }],
    "type": "visualization",
    "updated_at": "2020-11-25T15:09:25.722Z",
    "version": "WzIwLDFd"
}```

To get the auto-generated ID of the index pattern I am using

GET .kibana/_search
{
  "_source": ["index-pattern.title"],
  "query": {
    "term": {
      "type": "index-pattern"
    }
  }
}

Then I am using these IDs to update the dashboard so the visualizations reflect the new indices.

I wonder if this the right way to reuse the dashboard?
I inspected the dashboard attributes in the saved object, I only updated the dashboard name and the ID of referenced visualizations, but it did not work, are there other configurations I need to update for dashboard reuse.

Thanks

11

Hi again @flash1293, can you assist with this. Thanks.

12

Hey, sorry for the late reply.

The approach you describe sounds right to me - for each of the visualizations, you need to change the id of the referenced index pattern, for the dashboard you need to change the ids of the referenced visualizations.

What exactly is not working?

13

Hi again @flash1293

The dashboard and the visualisations have unique ids, how can I generate new ids so that when I duplicate the dashboards and the visualisations in my pipeline without a conflict. So the situation is, I copied the dashboard object using cp and I want to use the copied dashboard as a basis for the new one. I changed all the index patterns of the elastic indices, but the visualisations and the dashboard have autogenerated indices. How can I generate new indices, so I can reuse the same template of the dashboard and visualisation without using the UI.

14

You can pick anything to generate ids, there are no constraints, just make sure the references line up. It probably makes sense to use a uuid-generator. There are packages for all common languages (e.g. https://www.npmjs.com/package/uuid for node)

15

Thank you @flash1293. I used Python Library to generate uuid.

I still have an issue.

I only generate UUID for visualizations and dashboards, and then I did the mapping. For the index patterns of elastic indices, I did not generate them, instead, I extracted the uuid generated when I registered the patterns in kibana with

  curl -XPOST ${KIBANA}/api/saved_objects/index-pattern \
        -H "kbn-xsrf: true" \
        -H 'Content-Type: application/json;charset=UTF-8' \
        --data-binary "{\"attributes\":{\"title\":\"${index}\",\"timeFieldName\":\"${date}\"}}"

When I run the pipeline for the code, I got random results, sometimes I can generate two dashboards but most of the time I got an error. The error is linked to index patterns, I noticed that some index patterns are overwritten or lost. When I search in the output of the following call, I cannot find those patterns

curl -X GET "http://localhost:5601/api/saved_objects/_find?type=index-pattern&search_fields=title"

Not sure why this happens, is it linked to the uuid generated by kibana?

16

What's the error you are seeing?

17

Thank you @flash1293 for your responses throughout this thread.

The issue was:

  1. the exported dashboard seems to contain an entry of the default index pattern, along with the visualizations. This caused conflict when the dashboard is imported.
  2. Some index patterns are not listed when using this api call
    curl -X GET "http://localhost:5601/api/saved_objects/_find?type=index-pattern&search_fields=title" > $indx_patterns
    I have to add page=10000 to get all the index patterns.
    curl -s "${kibana_url}/api/saved_objects/_find?fields=title&fields=type&per_page=10000&type=index-pattern" > $indx_patterns

If you have time, I would appreciate your help in my other Kibana-vega issue

Thank you.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.