# Elasticsearch to elasticsearch index transfer

**URL:** <https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953>\
**Category:** Logstash\
**Created:** [May 15, 2018, 2:20pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953 "2018-05-15T14:20:45Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [May 15, 2018, 2:20pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953/1 "2018-05-15T14:20:45Z")

</div>

I am trying to move an index from my development cluster to our production cluster via logstash. I am getting the below response in the logs that looks like a web response. The development has no security so I'm not real sure what is answering. Advice would be appreciated.

input {  
elasticsearch {  
hosts =\> "135.89.18.199:9200"  
index =\> "apigw-example"  
query =\> '{ "query": { "match\_all": { } } }'  
}  
}  
filter {

}

output {  
stdout { codec =\> dots }  
elasticsearch {  
hosts =\> "[roacamu01.gcsc.att.com:9200](http://roacamu01.gcsc.att.com:9200)"  
user =\> "elastic"  
password =\> "xx"  
index =\> "goss-example-incident"  
document\_type =\> "%{[@metadata][\_type]}"  
document\_id =\> "%{[@metadata][\_id]}"  
}  
}

2018-05-15T08:52:41,550][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2018-05-15T08:52:44,425][WARN][logstash.agent] stopping pipeline {:id=\>"main"}  
[2018-05-15T08:54:00,276][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://elastic:xxxxxx@roacamu01.gcsc.att.com:9200/](http://elastic:xxxxxx@roacamu01.gcsc.att.com:9200/)]}}  
[2018-05-15T08:54:00,284][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://elastic:xxxxxx@roacamu01.gcsc.att.com:9200/](http://elastic:xxxxxx@roacamu01.gcsc.att.com:9200/), :path=\>"/"}  
[2018-05-15T08:54:00,694][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>#Java::JavaNet::URI:0x326b5725}  
[2018-05-15T08:54:00,697][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil}  
[2018-05-15T08:54:00,804][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
[2018-05-15T08:54:00,864][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::Elasticsearch", :hosts=\>[#Java::JavaNet::URI:0x45b05ea]}  
[2018-05-15T08:54:00,869][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500}  
[2018-05-15T08:54:00,916][INFO][logstash.pipeline] Pipeline main started  
[2018-05-15T08:54:01,058][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2018-05-15T08:54:01,105][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
Plugin: \<LogStash::Inputs::Elasticsearch hosts=\>["localhost:9200"], index=\>"apigw-example", query=\>"{ "query": { "match\_all": { } } }", id=\>"a33a4d8243e09141da783c0bd6b0f87a777f45d9-1", enable\_metric=\>true, codec=\>\<LogStash::Codecs::JSON id=\>"json\_c9b1a58f-2e22-4704-91aa-9f2497dc6d3e", enable\_metric=\>true, charset=\>"UTF-8"\>, size=\>1000, scroll=\>"1m", docinfo=\>false, docinfo\_target=\>"@metadata", docinfo\_fields=\>["\_index", "\_type", "\_id"], ssl=\>false\>  
Error: [407]

Access Denied  

> | Access Denied (authentication\_failed)  
>   
> |
> | Your credentials could not be authenticated: "Credentials are missing.". You will not be permitted access until your credentials can be verified. |
> | This is typically caused by an incorrect username and/or password, but could also be caused by network problems. |
> |   
> |

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2018, 2:53pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953/2 "2018-05-15T14:53:20Z")

</div>

The error message says it is talking to localhost:9200, which does not match the configuration you posted. Which is correct?

---

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [May 15, 2018, 4:39pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953/3 "2018-05-15T16:39:37Z")

</div>

Actually both. I tried with localhost and then with the ip address. Both do the same. I just got my paste mixed up. Sorry.

---

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [May 15, 2018, 5:25pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953/4 "2018-05-15T17:25:36Z")

</div>

I have restarted the conf and it ran for bit. Uploaded around 260,000 of around 6 million. At that point it started with the authorization again.

[2018-05-15T13:23:09,830][ERROR][logstash.pipeline] A plugin had an unre  
coverable error. Will restart this plugin.  
Plugin: \<LogStash::Inputs::Elasticsearch hosts=\>["135.89.18.199:9200"], index=

> "apigw-example", query=\>"{ "query": { "match\_all": { } } }", size=\>500, scr  
> oll=\>"30s", id=\>"98c338c98a064ea1057c8a48eac8166a04e36ba0-1", enable\_metric=\>tru  
> e, codec=\>\<LogStash::Codecs::JSON id=\>"json\_faaa54ff-fa52-49ec-8e9e-c9dd9565714e  
> ", enable\_metric=\>true, charset=\>"UTF-8"\>, docinfo=\>false, docinfo\_target=\>"@met  
> adata", docinfo\_fields=\>["\_index", "\_type", "\_id"], ssl=\>false\>  
> Error: [407]

Access Denied  

> | Access Denied (authentication\_failed)  
>   
> |
> | Your credentials could not be authenticated: "Credentials are missing.". You wil l not be permitted access until your credentials can be verified. |
> | This is typically caused by an incorrect username and/or password, but could als o be caused by network problems. |
> |   
> |

---

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [May 15, 2018, 5:27pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953/5 "2018-05-15T17:27:30Z")

</div>

I did modify the conf fiel with a size and scroll.

input {  
elasticsearch {  
hosts =\> "135.89.18.199:9200"  
index =\> "apigw-example"  
query =\> '{ "query": { "match\_all": { } } }'  
size =\> 500  
scroll =\> "30s"  
}  
}  
filter {

}

output {  
stdout { codec =\> dots }  
elasticsearch {  
hosts =\> "[roacamu01.gcsc.att.com:9200](http://roacamu01.gcsc.att.com:9200)"  
user =\> "elastic"  
password =\> "xx"  
index =\> "goss-example-incident"  
}  
}

---

<div class="post-metadata">

**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\
**Post date:** [May 15, 2018, 6:54pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953/6 "2018-05-15T18:54:32Z")

</div>

Shouldn't you add user and password to the input aswell? seems like to going wrong there...

---

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [May 16, 2018, 1:34pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953/7 "2018-05-16T13:34:29Z")

</div>

The input is off my development machine and it does not have security active. However, I have found the problem I believe. I am dealing the data across a subnet and I think the internal security is challenging me.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 13, 2018, 1:45pm UTC](https://discuss.elastic.co/t/elasticsearch-to-elasticsearch-index-transfer/131953/8 "2018-06-13T13:45:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.