Export and Import Index Patterns using elasticdump in ELK ver 7

[nguyenduyhung1979 (Hung Nguyen)]

I try setup automation system which has the workflow:

• Have server where ELK stack has been installed and has basic configuration
• Import the index into ElasticSearch (ES), next Kibana can use that index for Visualize

For preparing the input for the import index step, I did following steps to export index to json file

• Manually create index in ES
• Manually add that index into Kibana
• Using elasticdump commandline to export index pattern to json file:

elasticdump --input=http://localhost:9200 --input-index=.kibana/index-pattern --output=e:/ELK/dev.json --type=data

This commandline work well with ELK ver 5.6. But when I run it on ELK ver 7.11, I got empty json file. It's rather urgent for me now since I stuck here in 2 days. I appreciate any help here.

Thanks,

[stephenb (Stephen Brown)]

Hi @nguyenduyhung1979

If you look at the indices with

GET _cat/indices/*/?v

You will probably see that now it looks like

.kibana_1
.kibana_2
...

So you will probably need to use that, access to those indices is going to go away in the future.

That said the Correct way to export / import Kibana Objects is with the Saved Object API.

[nguyenduyhung1979 (Hung Nguyen)]

Hi Stephenb,

Based on page at: Export objects API | Kibana Guide [7.11] | Elastic I tried to use the following commandline to export ALL index:

curl -XPOST "localhost:5601/api/saved_objects/_export -H 'kbn-xsrf:true' -H 'Content-Type:application/json' -d '" -H 'kbn-xsrf:true' -H 'Content-Type:application/json' -d'{"type":"index-pattern"}'

But it didn't show me anything and it didn't throw any error. Could you help me a basic export index pattern curl commandline? I appreciate your help so much.

[stephenb (Stephen Brown)]

Your command is malformed.. you are missing a " after _export have an extra -d '" in the middle , just carefully copy from the examples.

Ahh it looks like the copy as curl link is broken!! adding the bad syntax!

So just highlight and copy

I ran...

curl -X POST http://localhost:5601/api/saved_objects/_export -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{"type": "index-pattern"}'

It worked fine to stdout

I ran this command to save to a file

curl -X POST http://localhost:5601/api/saved_objects/_export -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{"type": "index-pattern"}' > index-pattern.ndjson

[nguyenduyhung1979 (Hung Nguyen)]

It doesn't work for me

I have many indexes in system

[stephenb (Stephen Brown)]

Hi @nguyenduyhung1979

It is really hard for anyone to help / debug when you post a screen shot not the the text of the actual command you run.

Most likely your cut-n-paste put in smart quotes or something .. .the command is still malformed it is pretty clear that is is by the error message.... those single ticks should be single quotes not back quotes etc... looks like back quotes or something... not normal single quotes...

I just ran this command again works fine.

curl -X POST http://localhost:5601/api/saved_objects/_export -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{"type": "index-pattern"}'

Keep trying my command is correct ...

Also there is a difference between and index pattern and index you asked about index patterns at the the start of this thread. They are handled differently.... than normal indexes

@nguyenduyhung1979

Using elasticdump commandline to export index pattern to json file:

[nguyenduyhung1979 (Hung Nguyen)]

Hi Stephenb,

Yes, I understand index and index pattern are not the same and I need export index pattern which include all fields.

I re-typed the commandline you told me, I'm not copy-past. Here is my commandline:

curl -X POST http://localhost:5601/api/saved_objects/_export -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{"type": "index-pattern"}'

And here is the error I got:

{"statusCode":400,"error":"Bad Request","message":"Request must contain a kbn-xsrf header."}curl: (6) Could not resolve host: true'
curl: (6) Could not resolve host: application
curl: (3) [globbing] unmatched close brace/bracket in column 14

After I got that error, I try to remove space character after ":" character:

curl -X POST http://localhost:5601/api/saved_objects/_export -H 'kbn-xsrf:true' -H 'Content-Type:application/json' -d '{"type":"index-pattern"}'

And I got error:

{"statusCode":400,"error":"Bad Request","message":"Request must contain a kbn-xsrf header."}

I still don't know what's wrong here, my command is exactly yours, I re-tyoed it, not copy and past.

Regards,

[stephenb (Stephen Brown)]

@nguyenduyhung1979

curl -X POST http://localhost:5601/api/saved_objects/_export -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{"type": "index-pattern"}'

thanks,

Huh I run that ^^^^ your command, works for me on mac / linux

What version of elastic / kibana are you running that command against?

Also I noticed / I think you are on Windows.... perhaps curl is different on windows or you are on an old version?

what version of curl

curl --version

Also you can try double quote the url

curl -X POST "http://localhost:5601/api/saved_objects/_export" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{"type": "index-pattern"}'

Ahhhh looks like curl on windows might have problems with single quote!

[SOLVED] Curl(?) differences linux/windows

try this!! !

curl -X POST http://localhost:5601/api/saved_objects/_export -H "kbn-xsrf: true" -H "Content-Type: application/json" -d "{\"type\": \"index-pattern\"}"

[stephenb (Stephen Brown)]

Try this
curl -X POST http://localhost:5601/api/saved_objects/_export -H "kbn-xsrf: true" -H "Content-Type: application/json" -d "{\"type\": \"index-pattern\"}"

[nguyenduyhung1979 (Hung Nguyen)]

Yes, I run curl in Windowns

C:>curl --version
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL

YESSSSS !!! the command work for me .....

But the json output doesnot have field data. Here is output I got:

{"attributes":{"fieldAttrs":"{}","fields":"[]","timeFieldName":"@timestamp","title":"ps-marsha-rec-alarm-*"},"id":"0fa82b60-836f-11eb-b9c2-f77828f70718","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-03-12T20:10:59.222Z","version":"WzcyLDFd"}

{"attributes":{"fieldAttrs":"{}","fields":"[]","timeFieldName":"@timestamp","title":"ps-marsha-rec-perfmon-*"},"id":"5debfc80-836e-11eb-b9c2-f77828f70718","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-03-12T20:06:01.032Z","version":"WzQ0LDFd"}

{"exportedCount":2,"missingRefCount":0,"missingReferences":[]}

[nguyenduyhung1979 (Hung Nguyen)]

My index " ps-marsha-rec-alarm-*" has some fields: AlarmName, AlarmsNumber, ComponentName.

[stephenb (Stephen Brown)]

Ahhh another mystery... but actually it is not... index-patterns are built on the fly now from 7.11 and higher , so all the fields are not imported exported / imported, only if you have edited the fields properties to the right will they show up. This simplifies the whole index pattern capabilities.

You will notice in 7.11 on that there is no longer a index refresh it is automatic.

So only edits you have made will be exported, all normal fields in the index pattern will be built on the fly.

Also just FYI there were a few bugs in 7.11.0 I would upgrade to 7.11.2

So I tested I edited 1 field of many in the index pattern. and it showed up in my export, (That may have been one of the bugs fixed) So now only special / edited field formats etc need to be exported.

{"attributes":{"fieldAttrs":"{}","fieldFormatMap":"{\"name\":{\"id\":\"static_lookup\",\"params\":{\"lookupEntries\":[{\"key\":\"steve\",\"value\":\"old\"}]}}}","fields":"[]","title":"test"},

[nguyenduyhung1979 (Hung Nguyen)]

Excellent support ever!!!! ... I tried to edit 1 field and it show up in export data. It's late for me now (10pm). Tomorrow, I will try to import what I export today and I think it's easy since we've been thought many issue with export command already

Thanks and appreciate your help so so much,

[nguyenduyhung1979 (Hung Nguyen)]

Hi Stephenb,

I have a good news. I have 2 indexes (ps-marsha-rec-perfmon-index and ps-marsha-rec-alarm-index). I exported and imported the json file successfully but the Kibana didn't show up these index patterns. After I restarted Kibana service, these indexes show up correctly.

One more time, thanks for your help SO SO MUCH !!!!...

Here the import commandline I used:

curl -X POST "http://localhost:5601/api/saved_objects/_import" -H "kbn-xsrf: true" --form file=@F:\ELK\index-pattern.ndjson

Here what I did:

• Export index patterns into json file
• Clean up everything using commandline: curl -XDELETE "localhost:9200/_all"
• Import index pattern using the json above
• Run logstash to populate data into ES using same index name
• Restart Kibana service
• Kibana show up all my index patterns