Unable to copy data from one ES instance to another using LS

[crags (Craig Patrick)]

Hi.

I'm trying to copy some data from our production ES stack to our stage stack for testing using logstash, and for some reason it's just not working. I think it may be to do with the query, but I'm not sure for definite.

Below is my config I'm using (sanitised slightly):

input {
    elasticsearch {
        user => <user>
        password => <pass>
        hosts => <live hosts>
        index => "info-requests"

        docinfo => true
        query => '{ "query" : { "range": { "timestamp" : { "gte" : "now-1d/d", "lt" :"now/d" } } } }'
    }
}

## Remove fields added by logstash:

filter {
 mutate {
  remove_field => [ "@version", "@timestamp" ]
 }
}

# Output:
# We now shove all the yumminess into ElasticSearch, all being well!

output {
    elasticsearch {
        hosts => <stage hosts>

        manage_template => false

        index => "%{[@metadata][_index]}"
        document_type => "%{[@metadata][_type]}"
        document_id => "%{[@metadata][_id]}"

    }

    stdout { codec => dots }

}

Now if I remove the query it starts to write data into the stage ES stack, but ALL the data which I don't want. If I take the query and run it on our production stack to check records I get the following results:

Query:

GET /info-requests/_search
{ "query" : { "range": { "timestamp" : { "gte" : "now-1d/d", "lt" :"now/d" } } } }

Results:

"took": 81,
  "timed_out": false,
  "_shards": {
    "total": 222,
    "successful": 222,
    "failed": 0
  },
  "hits": {
    "total": 24864275,
    "max_score": 1,
    "hits": [
  .... etc ....

so I know the query should be good but it seems as if it's not returning any results when running in Logstash or something? Under --debug mode I get the following at the end (I wont put it all here):

16:02:19.762 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
16:02:19.844 [Ruby-0-Thread-9: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:489] DEBUG logstash.pipeline - Pushing flush onto pipeline
16:02:19.847 [[main]-pipeline-manager] DEBUG logstash.pipeline - Pushing shutdown {:thread=>"#<Thread:0x59763f17 run>"}
16:02:19.847 [[main]-pipeline-manager] DEBUG logstash.pipeline - Pushing shutdown {:thread=>"#<Thread:0x7e978250 sleep>"}
16:02:19.848 [[main]-pipeline-manager] DEBUG logstash.pipeline - Shutdown waiting for worker thread #<Thread:0x59763f17>
16:02:19.857 [[main]-pipeline-manager] DEBUG logstash.pipeline - Shutdown waiting for worker thread #<Thread:0x7e978250>
16:02:19.858 [[main]-pipeline-manager] DEBUG logstash.filters.mutate - closing {:plugin=>"LogStash::Filters::Mutate"}
16:02:19.858 [[main]-pipeline-manager] DEBUG logstash.outputs.elasticsearch - closing {:plugin=>"LogStash::Outputs::ElasticSearch"}
16:02:19.858 [[main]-pipeline-manager] DEBUG logstash.outputs.stdout - closing {:plugin=>"LogStash::Outputs::Stdout"}
16:02:19.858 [[main]-pipeline-manager] DEBUG logstash.pipeline - Pipeline main has been shutdown
16:02:20.558 [pool-2-thread-1] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-03-23 16:02:20 +0000}
16:02:21.570 [pool-2-thread-1] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-03-23 16:02:21 +0000}
16:02:22.579 [pool-2-thread-1] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-03-23 16:02:22 +0000}
16:02:22.734 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.os - PeriodicPoller: Stopping
16:02:22.735 [LogStash::Runner] DEBUG logstash.instrument.periodicpoller.jvm - PeriodicPoller: Stopping
16:02:22.744 [LogStash::Runner] WARN  logstash.agent - stopping pipeline {:id=>"main"}
16:02:22.745 [LogStash::Runner] DEBUG logstash.pipeline - Closing inputs
16:02:22.746 [LogStash::Runner] DEBUG logstash.inputs.elasticsearch - stopping {:plugin=>"LogStash::Inputs::Elasticsearch"}
16:02:22.746 [LogStash::Runner] DEBUG logstash.pipeline - Closed inputs

and still nothing in the stage ES stack.

All help greatly appreciated

thanks!