Need help: Can't migrate indices between clusters :(


(elasticsearch) #1

The situation:

First elasticsearch cluster:
{
"cluster_name" : "jscluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 10,
"active_shards" : 20,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

that cluster contain 2 indices: vit and usr

Target: copy all indices to other elasticsearch cluster:
{
"cluster_name" : "js-t100",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

How to do for copy all indices from jscluster to js-t100?

What I'm already try to do:
On js-t100:

  1. curl -o /tmp/js_vit -XGET
    'http://jscluster.test:9200/vit/_mapping?pretty=true'
  2. sed -i '1,2d' /tmp/js_vit
  3. echo
    '{"settings":{"number_of_shards":5,"number_of_replicas":1},"mappings":
    {'

/tmp/mappost

  1. cat /tmp/js_vit >>/tmp/mappost
  2. curl -XPUT 'http://js-t100.test:9200/vit/' -d 'cat /tmp/mappost'
  3. stop js-t100 elasticsearch
  4. copy indices/vit/0..4 from jscluster nodes into
    data/js-t100/nodes/0/indices/vit/
  5. start elasticsearch on js-t100.test
  6. check health of js-t100.test cluster - unsaccess
    In log i get the messages:

[2011-11-07 11:47:57,222][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: initializing ...
[2011-11-07 11:47:57,225][INFO ][plugins ] [Peace
Monger] loaded [], sites []
[2011-11-07 11:47:58,639][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: initialized
[2011-11-07 11:47:58,640][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: starting ...
[2011-11-07 11:47:58,686][INFO ][transport ] [Peace
Monger] bound_address {inet[/10.10.240.7:9300]}, publish_address
{inet[/10.10.240.7:9300]}
[2011-11-07 11:48:01,712][INFO ][cluster.service ] [Peace
Monger] new_master [Peace
Monger][9K08oguqSAC4QpRwogHk9Q][inet[/10.10.240.7:9300]],
reason:zen-disco-join (elected_as_master)
[2011-11-07 11:48:01,740][INFO ][discovery ] [Peace
Monger] js-t100/9K08oguqSAC4QpRwogHk9Q
[2011-11-07 11:48:01,751][INFO ][http ] [Peace
Monger] bound_address {inet[/10.10.240.7:9200]}, publish_address
{inet[/10.10.240.7:9200]}
[2011-11-07 11:48:01,752][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: started
[2011-11-07 11:48:01,833][INFO ][gateway ] [Peace
Monger] recovered [0] indices into cluster_state
[2011-11-07 11:54:46,668][INFO ][cluster.metadata ] [Peace
Monger] [vit] creating index, cause [api], shards [5]/[1], mappings []
[2011-11-07 11:55:23,424][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: stopping ...
[2011-11-07 11:55:23,445][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: stopped
[2011-11-07 11:55:23,445][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: closing ...
[2011-11-07 11:55:23,452][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: closed

[2011-11-07 11:57:09,418][INFO ][bootstrap ]
max_open_files [99973]
[2011-11-07 11:57:09,454][INFO ][node ]
[Architect]
{0.18.2}[17902]: initializing ...
[2011-11-07 11:57:09,465][INFO ][plugins ]
[Architect]
loaded [], sites []
[2011-11-07 11:57:11,152][INFO ][node ]
[Architect]
{0.18.2}[17902]: initialized
[2011-11-07 11:57:11,153][INFO ][node ]
[Architect]
{0.18.2}[17902]: starting ...
[2011-11-07 11:57:11,200][INFO ][transport ]
[Architect]
bound_address {inet[/10.10.240.7:9300]}, publish_address
{inet[/10.10.240.7:9300]}
[2011-11-07 11:57:14,227][INFO ][cluster.service ]
[Architect]
new_master [Architect][9olcJlaBQiWXwRnu8hsWow][inet[/
10.10.240.7:9300]],
reason: zen-disco-join (elected_as_master)
[2011-11-07 11:57:14,265][INFO ][discovery ]
[Architect]
js-t100/9olcJlaBQiWXwRnu8hsWow
[2011-11-07 11:57:14,281][INFO ][http ]
[Architect]
bound_address {inet[/10.10.240.7:9200]}, publish_address
{inet[/10.10.240.7:9200]}
[2011-11-07 11:57:14,282][INFO ][node ]
[Architect]
{0.18.2}[17902]: started
[2011-11-07 11:57:14,700][INFO ][gateway ]
[Architect]
recovered [1] indices into cluster_state


And answer on cluster's health request:

curl -XGET 'http://js-t100.test:9200/_cluster/health?pretty=true'
{
"cluster_name" : "js-t100",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5
}

And status of the cluster stay yellow constantely O'course I get the
wrong answer on request like that:

curl -XPOST http://js-t100.test:9200/vit/_search
{"error":"ActionRequestValidationException[Validation Failed: 1:
search
source is missing;]","status":500}

Last 5 days I can't understand what is wrong I do and I don't know how
I can
migrate data between clusters Heeeeeelp! S.O.S.


(Karussell) #2

I would go with the scan search.

Peter.

On Nov 7, 11:53 am, elasticsearch pentagram...@rambler.ru wrote:

The situation:

First elasticsearch cluster:
{
"cluster_name" : "jscluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 10,
"active_shards" : 20,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0

}

that cluster contain 2 indices: vit and usr

Target: copy all indices to other elasticsearch cluster:
{
"cluster_name" : "js-t100",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0

}

How to do for copy all indices from jscluster to js-t100?

What I'm already try to do:
On js-t100:

  1. curl -o /tmp/js_vit -XGET
    'http://jscluster.test:9200/vit/_mapping?pretty=true'

  2. sed -i '1,2d' /tmp/js_vit

  3. echo
    '{"settings":{"number_of_shards":5,"number_of_replicas":1},"mappings":
    {'>> /tmp/mappost

  4. cat /tmp/js_vit >>/tmp/mappost

  5. curl -XPUT 'http://js-t100.test:9200/vit/'-d 'cat /tmp/mappost'

  6. stop js-t100 elasticsearch

  7. copy indices/vit/0..4 from jscluster nodes into
    data/js-t100/nodes/0/indices/vit/

  8. start elasticsearch on js-t100.test

  9. check health of js-t100.test cluster - unsaccess
    In log i get the messages:


[2011-11-07 11:47:57,222][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: initializing ...
[2011-11-07 11:47:57,225][INFO ][plugins ] [Peace
Monger] loaded [], sites []
[2011-11-07 11:47:58,639][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: initialized
[2011-11-07 11:47:58,640][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: starting ...
[2011-11-07 11:47:58,686][INFO ][transport ] [Peace
Monger] bound_address {inet[/10.10.240.7:9300]}, publish_address
{inet[/10.10.240.7:9300]}
[2011-11-07 11:48:01,712][INFO ][cluster.service ] [Peace
Monger] new_master [Peace
Monger][9K08oguqSAC4QpRwogHk9Q][inet[/10.10.240.7:9300]],
reason:zen-disco-join (elected_as_master)
[2011-11-07 11:48:01,740][INFO ][discovery ] [Peace
Monger] js-t100/9K08oguqSAC4QpRwogHk9Q
[2011-11-07 11:48:01,751][INFO ][http ] [Peace
Monger] bound_address {inet[/10.10.240.7:9200]}, publish_address
{inet[/10.10.240.7:9200]}
[2011-11-07 11:48:01,752][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: started
[2011-11-07 11:48:01,833][INFO ][gateway ] [Peace
Monger] recovered [0] indices into cluster_state
[2011-11-07 11:54:46,668][INFO ][cluster.metadata ] [Peace
Monger] [vit] creating index, cause [api], shards [5]/[1], mappings []
[2011-11-07 11:55:23,424][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: stopping ...
[2011-11-07 11:55:23,445][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: stopped
[2011-11-07 11:55:23,445][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: closing ...
[2011-11-07 11:55:23,452][INFO ][node ] [Peace
Monger] {0.18.2}[17227]: closed

[2011-11-07 11:57:09,418][INFO ][bootstrap ]
max_open_files [99973]
[2011-11-07 11:57:09,454][INFO ][node ]
[Architect]
{0.18.2}[17902]: initializing ...
[2011-11-07 11:57:09,465][INFO ][plugins ]
[Architect]
loaded [], sites []
[2011-11-07 11:57:11,152][INFO ][node ]
[Architect]
{0.18.2}[17902]: initialized
[2011-11-07 11:57:11,153][INFO ][node ]
[Architect]
{0.18.2}[17902]: starting ...
[2011-11-07 11:57:11,200][INFO ][transport ]
[Architect]
bound_address {inet[/10.10.240.7:9300]}, publish_address
{inet[/10.10.240.7:9300]}
[2011-11-07 11:57:14,227][INFO ][cluster.service ]
[Architect]
new_master [Architect][9olcJlaBQiWXwRnu8hsWow][inet[/
10.10.240.7:9300]],
reason: zen-disco-join (elected_as_master)
[2011-11-07 11:57:14,265][INFO ][discovery ]
[Architect]
js-t100/9olcJlaBQiWXwRnu8hsWow
[2011-11-07 11:57:14,281][INFO ][http ]
[Architect]
bound_address {inet[/10.10.240.7:9200]}, publish_address
{inet[/10.10.240.7:9200]}
[2011-11-07 11:57:14,282][INFO ][node ]
[Architect]
{0.18.2}[17902]: started
[2011-11-07 11:57:14,700][INFO ][gateway ]
[Architect]
recovered [1] indices into cluster_state


And answer on cluster's health request:

curl -XGET 'http://js-t100.test:9200/_cluster/health?pretty=true'
{
"cluster_name" : "js-t100",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5}


And status of the cluster stay yellow constantely O'course I get the
wrong answer on request like that:

curl -XPOSThttp://js-t100.test:9200/vit/_search
{"error":"ActionRequestValidationException[Validation Failed: 1:
search
source is missing;]","status":500}

Last 5 days I can't understand what is wrong I do and I don't know how
I can
migrate data between clusters Heeeeeelp! S.O.S.


(elasticsearch) #3

Can you explain it to me a little bit more?
May be several snippets?
I want to create easy shell-script for migrating.

On Nov 7, 7:52 pm, Karussell tableyourt...@googlemail.com wrote:

I would go with the scan search.

Peter.


(system) #4