Hi, I have a very small ELK cluster with about 1.6B documents in about 500GB.
I need to export this data to another much larger ELK cluster.
Are there any documents which describe this and/or tips you can give?
Thanks!
You can try my import/export plugin https://github.com/jprante/elasticsearch-knapsack
Thanks I tried it but am given the error "document is empty" on the test cluster.
scracraft@tstes1:~$ /usr/share/elasticsearch/bin/plugin --list
Installed plugins:
- HQ
- head
- knapsack
- marvel
- bigdesk
scracraft@tstes1:~$ uptime
21:44:14 up 22 min, 1 user, load average: 0.30, 0.19, 0.11 <= node was rebooted and in green state
scracraft@tstes1:~$ source ~/.aliases
scracraft@tstes1:~$ indices | grep test
scracraft@tstes1:~$ curl -XDELETE localhost:9200/test
{"error":"RemoteTransportException[[tstes5][inet[/10.10.5.165:9300]][indices:admin/delete]]; nested: IndexMissingException
[[test] missing]; ","status":404}scracraft@tstes1:~$
scracraft@tstes1:~$ curl -XPUT localhost:9200/test/test/1 -d '{"key":"value 1"}'
{"_index":"test","_type":"test","_id":"1","_version":1,"created":true}scracraft@tstes1:~$ indices | grep test
"name": "test"
scracraft@tstes1:~$ indices | tail
"name": ".marvel-2016.07.12"
}
{
"count": 1,
"name": ".marvel-kibana"
}
{
"count": 1,
"name": "test"
}
scracraft@tstes1:~$ curl -XPUT localhost:9200/test/test/2 -d '{"key":"value 2"}'
{"_index":"test","_type":"test","_id":"2","_version":1,"created":true}scracraft@tstes1:~$ indices | tail
"name": ".marvel-2016.07.12"
}
{
"count": 1,
"name": ".marvel-kibana"
}
{
"count": 2,
"name": "test"
}
scracraft@tstes1:~$ curl -XPOST localhost:9200/test/test/_export
{"error":"RemoteTransportException[[tstes5][inet[/10.10.5.165:9300]][indices:data/write/index]]; nested: MapperParsingExce
ption[failed to parse, document is empty]; ","status":400}scracraft@tstes1:~$
scracraft@tstes1:~$
Snapshot and restore.
Ls, like this https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06
What ES version do you use?
Here it works:
$ curl -XPUT localhost:9200/test/test/1 -d '{"key":"value 1"}'
{"_index":"test","_type":"test","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}
$ curl -XPOST localhost:9200/test/test/_export
{"running":true,"state":{"mode":"export","started":"2016-07-12T22:34:18.355Z","path":"file:///Users/es/elasticsearch-2.3.3/logs/test_test.tar.gz","node_name":"Soulfire"}}
Logfile
[2016-07-13 00:34:02,323][INFO ][cluster.metadata ] [Soulfire] [test] update_mapping [test]
[2016-07-13 00:34:18,356][INFO ][org.xbib.elasticsearch.action.knapsack.exp.TransportKnapsackExportAction] [Soulfire] start of export: {"mode":"export","started":"2016-07-12T22:34:18.355Z","path":"file:///Users/es/elasticsearch-2.3.3/logs/test_test.tar.gz","node_name":"Soulfire"}
[2016-07-13 00:34:18,364][INFO ][cluster.metadata ] [Soulfire] [.knapsack] creating index, cause [api], templates [], shards [5]/[1], mappings []
[2016-07-13 00:34:18,413][INFO ][cluster.routing.allocation] [Soulfire] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.knapsack][4]] ...]).
[2016-07-13 00:34:48,426][INFO ][cluster.metadata ] [Soulfire] [.knapsack] create_mapping [knapsack]
[2016-07-13 00:34:48,472][INFO ][org.xbib.elasticsearch.action.knapsack.exp.TransportKnapsackExportAction] [Soulfire] getting settings for indices [test]
[2016-07-13 00:34:48,473][INFO ][org.xbib.elasticsearch.action.knapsack.exp.TransportKnapsackExportAction] [Soulfire] found indices: [test]
[2016-07-13 00:34:48,475][INFO ][org.xbib.elasticsearch.action.knapsack.exp.TransportKnapsackExportAction] [Soulfire] getting mappings for index test and types [test]
[2016-07-13 00:34:48,476][INFO ][org.xbib.elasticsearch.action.knapsack.exp.TransportKnapsackExportAction] [Soulfire] found mappings: [test]
[2016-07-13 00:34:48,476][INFO ][org.xbib.elasticsearch.action.knapsack.exp.TransportKnapsackExportAction] [Soulfire] adding mapping: test
[2016-07-13 00:34:48,476][INFO ][org.xbib.elasticsearch.action.knapsack.exp.TransportKnapsackExportAction] [Soulfire] getting aliases for index test
[2016-07-13 00:34:48,477][INFO ][org.xbib.elasticsearch.action.knapsack.exp.TransportKnapsackExportAction] [Soulfire] found 0 aliases
[2016-07-13 00:34:48,517][INFO ][org.xbib.elasticsearch.action.knapsack.exp.TransportKnapsackExportAction] [Soulfire] end of export: {"mode":"export","started":"2016-07-12T22:34:18.355Z","path":"file:///Users/es/elasticsearch-2.3.3/logs/test_test.tar.gz","node_name":"Soulfire"}, packets = 3, docs = 1, total bytes transferred = 0, rate = 0,000000
Feel free to open an issue at https://github.com/jprante/elasticsearch-knapsack/issues with error messages from the server logfile.
On the test ES cluster, 1.6.2. On the production cluster 1.6.0.
As we don't want to expand anything for these, we are unlikely to upgrade.
Our goal is just to get all the data from the production cluster and upload it to a different ES cluster
than the above two.
I have to prepare a backport of knapsack to pre 2.0 versions.
By all means, do not use 1.6.0 in production. E.g.
Elasticsearch versions prior to 1.6.1 are vulnerable to an engineered attack on its transport protocol (used for communication between nodes and Java clients) that enables remote code execution.
Hi Jorg,
First, thanks ahead for the backport of knapsack to pre-2.0. We definitely need it.
A couple of questions:
- We had a rush launch and are committed to 1.6.0 in production now until
the data can be taken out of the cluster and put in a 2.x cluster which exists elsewhere.
In either event, we’ll need the backport of knapsack to pre 2.0 to escape 1.6.0
Because we are now in production, I won’t be upgrading the 1.6.0 cluster to 2.x as it would
present too much danger to production. Things are running perfectly and I don’t want to disturb
the checksum with changes that aren’t necessary.
Another question:
- what is so bad about 1.6.0?
Cheers,
—Stuart
Any news on backporting knapsack to 1.6.0 jprante? We'd like to get at that data in there and not just throw it all away.
Jorg: Any news? --Stuart
Backporting current knapsack to ES 1.6.0 is in progress, it's a lot of work.
In addition to the warning I quoted, 1.6.0 is mentioned in the https://www.elastic.co/community/security page, this version suffers from known vulnerability.
Okay, thanks for your previous comments. They are most helpful. Most helpful.