Curl'ing Kibana Dashboards exported into NDJSON

plissken · February 8, 2023, 11:44am

I'm having considerable difficulty in understanding the documentation on the Kibana API's. There's so much out of date information on the web that I could literally spend months doing syntax jenga. I have a set of dashboards and visualisations exported from the kibana dashboard into an ndjson file. I'm trying to curl this file back into production systems via the kibana dashboards/import API. My base command looks like this ...

curl -k -XPOST -H 'kbn-xsrf: true' http://127.0.0.1:5601/s/test/api/kibana/dashboards/import -d @dashboardfile.ndjson

I've tried several syntax changes, which always result in errors. Currently getting a 404
"error":"Bad Request","message":"[request body.objects]: expected value of type [array] but got [undefined]"}

Could someone kindly assist before I'm escorted into a padded cell? Thank you in advance.

stephenb · February 8, 2023, 2:48pm

What version are you on?

I assume you actually have a space named test

I Recommend sticking with our documentation.

Tell me the version and I will test

Assuming you are using a fairly new version you should use the Saved Objects API

I suspect your error is that your ndjson is malformed it needs to be an array so curious how did you create / download the dashboardfile.ndjson

plissken · February 8, 2023, 9:39pm

Hi Stephen. I'm on 8.5.0. The ndjson file was created by Kibana on the export saved objects menu. So I went into the kibana UI, navigated to Kibana->Saved Objects-> and exported the selected dashboard. Included all objects associated with the dashboard. Haven't touched the ndjson file, so trying to curl it as downloaded. I did try some permutations of curl for saved objects API but couldn't get that working either. Thanks.

stephenb · February 8, 2023, 9:56pm

Pretty sure you should then be using the saved object API not the dashboard API to reload.

You are mixing UI and APIs.

I will check a little later if I can

plissken · February 9, 2023, 8:09am

Thanks Stephen. I also missed getting back to you about the space "test". Yes, it's been created and ready for dashboards.

plissken · February 10, 2023, 2:11pm

Hi Stephen. Similar issues using the saved objects API. I've tried the following:

curl -k -XPOST -H 'kbn-xsrf: true' -H 'Content-Type: application/json'
http://127.0.0.1:5601/s/test/api/saved_objects/_bulk_create -d @app-server-system-overview.ndjso

which returns {"statusCode":400,"error":"Bad Request","message":"Invalid request payload JSON format"}

and trying x-ndjson (thinking it may explain the error)

{"statusCode":415,"error":"Unsupported Media Type","message":"Unsupported Media Type"}

And then avoiding the bulk API in favour of saved objects/dashboards

curl -k -XPOST -H 'kbn-xsrf: true' -H 'Content-Type: application/json'
http://127.0.0.1:5601/s/test/api/saved_objects/dashboard -d @app-server-system-overview.ndjson

I get {"statusCode":400,"error":"Bad Request","message":"Invalid request payload JSON format"}

There must be something wrong with the ndjson file. Perhaps I need to pre-format it before importing it. But as it comes directly from the Kibana UI export I didn't think this was necessary. Extract of app-server-system-overview.ndjson as follows:

{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{"query":{"query":"","language":"kuery"},"filter":}"},"title":"CPU Usage [Metricbeat System] ECS","uiStateJSON":"{}","version":1,"visState":"{"title":"CPU Usage [Metricbeat System] ECS","type":"metrics","aggs":,"params":{"time_range_mode":"entire_time_range","axis_formatter":"number","axis_position":"left","id":"80a04950-1b19-11e7-b09e-037021c4f8df","index_pattern":"metricbeat-*","interval":"auto","series":[{"time_range_mode":"entire_time_range","axis_position":"right","chart_type":"line","color":"#68BC00","fill":"1","formatter":"percent","id":"80a04951-1b19-11e7-b09e-037021c4f8df","label":"user","line_width":1,"metrics":[{"field":"system.cpu.user.pct","id":"80a04952-1b19-11e7-b09e-037021c4f8df","type":"avg"}],"point_size":"0","seperate_axis":0,"split_color_mode":"gradient","split_mode":"everything","stacked":"stacked"},{"time_range_mode":"entire_time_range","axis_position":"right","chart_type":"line","color":"rgba(211,49,21,1)","fill":"1","formatter":"percent","id":"993acf30-1b19-11e7-b09e-037021c4f8df","label":"system","line_width":1,"metrics":[{"field":"system.cpu.system.pct","id":"993acf31-1b19-11e7-b09e-037021c4f8df","type":"avg"}],"point_size":"0","seperate_axis":0,"split_color_mode":"gradient","split_mode":"everything","stacked":"stacked"},{"time_range_mode":"entire_time_range","axis_position":"right","chart_type":"line","color":"rgba(123,100,255,1)","fill":"1","formatter":"percent","id":"65ca35e0-1b1a-11e7-b09e-037021c4f8df","label":"nice","line_width":1,"metrics":[{"field":"system.cpu.nice.pct","id":"65ca5cf0-1b1a-11e7-b09e-037021c4f8df","type":"avg"}],"point_size":"0","seperate_axis":0,"split_color_mode":"gradient","split_mode":"everything","stacked":"stacked"},{"time_range_mode":"entire_time_range","axis_position":"right","chart_type":"line","color":"rgba(226,115,0,1)","fill":"1","formatter":"percent","id":"741b5f20-1b1a-11e7-b09e-037021c4f8df","label":"irq","line_width":1,"metrics":[{"field":"system.cpu.irq.pct","id":"741b5f21-1b1a-11e7-b09e-037021c4f8df","type":"avg"}],"point_size":"0","seperate_axis":0,"split_color_mode":"gradient","split_mode":"everything","stacked":"stacked"},{"time_range_mode":"entire_time_range","axis_position":"right","chart_type":"line","color":"rgba(176,188,0,1)","fill":"1","formatter":"percent","id":"2efc5d40-1b1a-11e7-b09e-037021c4f8df","label":"softirq","line_width":1,"metrics":[{"field":"system.cpu.softirq.pct","id":"2efc5d41-1b1a-11e7-b09e-037021c4f8df","type":"avg"}],"point_size":"0","seperate_axis":0,"split_color_mode":"gradient","split_mode":"everything","stacked":"stacked"},{"time_range_mode":"entire_time_range","axis_position":"right","chart_type":"line","color":"rgba(15,20,25,1)","fill":"1","formatter":"percent","id":"ae644a30-1b19-11e7-b09e-037021c4f8df","label":"iowait","line_width":1,"metrics":[{"field":"system.cpu.iowait.pct","id":"ae644a31-1b19-11e7-b09e-037021c4f8df","type":"avg"}],"point_size":"0","seperate_axis":0,"split_color_mode":"gradient","split_mode":"everything","stacked":"stacked"}],"show_grid":1,"show_legend":1,"time_field":"@timestamp","type":"timeseries","use_kibana_indexes":false,"drop_last_bucket":1,"axis_scale":"normal","truncate_legend":1,"max_lines_legend":1,"tooltip_mode":"show_all","isModelInvalid":false,"filter":{"query":"host.name:\"metricbeat-pri\"","language":"kuery"}}}"},"coreMigrationVersion":"8.5.0","id":"79e5b8fd-704b-4e00-a38f-6f35e6914773","migrationVersion":{"visualization":"8.5.0"},"references":,"type":"visualization","updated_at":"2023-02-08T08:24:33.365Z","version":"Wzk1MDEsM10="}
{"attributes"

stephenb · February 10, 2023, 4:34pm

Ok @plissken

Here is what I have for you now... and Wow that seemed a bit harder than I thought ...

So this is for an individual I used _export and _import ... I have not tried _bulk yet

So first I list to find the Object I am after in my initial / default space

# Get Dashboards
curl -X GET -H 'Content-Type: application/json' -H 'kbn-xsrf: true' http://localhost:5601/api/saved_objects/_find?type=dashboard | jq

In this case is is the eCommerce Dashboard

    {
      "type": "dashboard",
      "id": "722b74f0-b882-11e8-a6d9-e546fe2bba5f",

So now I _export it ... it takes both the type and id as far as I can tell in the objects array

# Export the actual Dashboard
curl -X POST -H 'Content-Type: application/json' -H 'kbn-xsrf: true' http://localhost:5601/api/saved_objects/_export -d '{"objects":[{"type":"dashboard","id":"722b74f0-b882-11e8-a6d9-e546fe2bba5f"}]}' > saved-dashboard.ndjson

Now _import to another space look at the syntax careful

# Import to a new space
curl -X POST -H 'kbn-xsrf: true' http://localhost:5601/s/test/api/saved_objects/_import?createNewCopies=true --form file=@saved-dashboard.ndjson
{"successCount":1,"success":true,"warnings":[],"successResults":[{"type":"dashboard","id":"722b74f0-b882-11e8-a6d9-e546fe2bba5f","meta":{"title":"[eCommerce] Revenue Dashboard","icon":"dashboardApp"},"destinationId":"e410188d-e9fd-40cc-bdc9-aa307d0a7165"}]}

And my Pretty new dashboard in my space test

I can probably look at the _bulk at some point if you still need that

plissken · February 10, 2023, 4:53pm

That's great Stephen. Thanks for all the hard work on this. Feels like an individual object export and import might be the best way forward. I'll have a play over the weekend. Have a good one.

stephenb · February 10, 2023, 4:55pm

Its bugging me... I will probably try the _bulk at some point

system · March 10, 2023, 4:56pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.