# Unable to copy data from one ES instance to another using LS

**URL:** <https://discuss.elastic.co/t/unable-to-copy-data-from-one-es-instance-to-another-using-ls/79772>\
**Category:** Logstash\
**Created:** [March 23, 2017, 4:05pm UTC](https://discuss.elastic.co/t/unable-to-copy-data-from-one-es-instance-to-another-using-ls/79772 "2017-03-23T16:05:10Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![crags](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crags/32/15112_2.png) [@crags](https://discuss.elastic.co/u/crags)\
**Post date:** [March 23, 2017, 4:05pm UTC](https://discuss.elastic.co/t/unable-to-copy-data-from-one-es-instance-to-another-using-ls/79772/1 "2017-03-23T16:05:10Z")

</div>

Hi.

I'm trying to copy some data from our `production` ES stack to our `stage` stack for testing using logstash, and for some reason it's just not working. I think it may be to do with the query, but I'm not sure for definite.

Below is my config I'm using (sanitised slightly):

```auto
input {
    elasticsearch {
        user => <user>
        password => <pass>
        hosts => <live hosts>
        index => "info-requests"

        docinfo => true
        query => '{ "query" : { "range": { "timestamp" : { "gte" : "now-1d/d", "lt" :"now/d" } } } }'
    }
}

## Remove fields added by logstash:

filter {
 mutate {
  remove_field => ["@version", "@timestamp"]
 }
}

# Output:
# We now shove all the yumminess into ElasticSearch, all being well!

output {
    elasticsearch {
        hosts => <stage hosts>

        manage_template => false

        index => "%{[@metadata][_index]}"
        document_type => "%{[@metadata][_type]}"
        document_id => "%{[@metadata][_id]}"

    }

    stdout { codec => dots }

}

```

Now if I remove the query it starts to write data into the stage ES stack, but ALL the data which I don't want. If I take the query and run it on our production stack to check records I get the following results:

Query:

```auto
GET /info-requests/_search
{ "query" : { "range": { "timestamp" : { "gte" : "now-1d/d", "lt" :"now/d" } } } }

```

Results:

```auto
"took": 81,
  "timed_out": false,
  "_shards": {
    "total": 222,
    "successful": 222,
    "failed": 0
  },
  "hits": {
    "total": 24864275,
    "max_score": 1,
    "hits": [
  .... etc ....

```

so I know the query should be good but it seems as if it's not returning any results when running in Logstash or something? Under `--debug` mode I get the following at the end (I wont put it all here):

```auto
16:02:19.762 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
16:02:19.844 [Ruby-0-Thread-9: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:489] DEBUG logstash.pipeline - Pushing flush onto pipeline
16:02:19.847 [[main]-pipeline-manager] DEBUG logstash.pipeline - Pushing shutdown {:thread=>"#<Thread:0x59763f17 run>"}
16:02:19.847 [[main]-pipeline-manager] DEBUG logstash.pipeline - Pushing shutdown {:thread=>"#<Thread:0x7e978250 sleep>"}
16:02:19.848 [[main]-pipeline-manager] DEBUG logstash.pipeline - Shutdown waiting for worker thread #<Thread:0x59763f17>
16:02:19.857 [[main]-pipeline-manager] DEBUG logstash.pipeline - Shutdown waiting for worker thread #<Thread:0x7e978250>
16:02:19.858 [[main]-pipeline-manager] DEBUG logstash.filters.mutate - closing {:plugin=>"LogStash::Filters::Mutate"}
16:02:19.858 [[main]-pipeline-manager] DEBUG logstash.outputs.elasticsearch - closing {:plugin=>"LogStash::Outputs::ElasticSearch"}
16:02:19.858 [[main]-pipeline-manager] DEBUG logstash.outputs.stdout - closing {:plugin=>"LogStash::Outputs::Stdout"}
16:02:19.858 [[main]-pipeline-manager] DEBUG logstash.pipeline - Pipeline main has been shutdown
16:02:20.558 [pool-2-thread-1] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-03-23 16:02:20 +0000}
16:02:21.570 [pool-2-thread-1] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-03-23 16:02:21 +0000}
16:02:22.579 [pool-2-thread-1] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-03-23 16:02:22 +0000}
16:02:22.734 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.os - PeriodicPoller: Stopping
16:02:22.735 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.jvm - PeriodicPoller: Stopping
16:02:22.744 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>"main"}
16:02:22.745 [LogStash::Runner] DEBUG logstash.pipeline - Closing inputs
16:02:22.746 [LogStash::Runner] DEBUG logstash.inputs.elasticsearch - stopping {:plugin=>"LogStash::Inputs::Elasticsearch"}
16:02:22.746 [LogStash::Runner] DEBUG logstash.pipeline - Closed inputs

```

and still nothing in the stage ES stack.

All help greatly appreciated 🙂

thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 20, 2017, 4:05pm UTC](https://discuss.elastic.co/t/unable-to-copy-data-from-one-es-instance-to-another-using-ls/79772/2 "2017-04-20T16:05:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.