# Logstash "elasticsearch input" problem

**URL:** <https://discuss.elastic.co/t/logstash-elasticsearch-input-problem/86847>\
**Category:** Logstash\
**Created:** [May 23, 2017, 5:52pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-problem/86847 "2017-05-23T17:52:22Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![orhiee](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@orhiee](https://discuss.elastic.co/u/orhiee)\
**Post date:** [May 23, 2017, 5:52pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-problem/86847/1 "2017-05-23T17:52:22Z")

</div>

hello,

Couldnt find the answer on here or google, so here i am asking....

i want to use logstash to transfer data from one elasticsearch to another, but i cant move all the docs can only move the number spesified in the "size" parameter:

```
input {
 # Read all documents from Elasticsearch matching the given query
  elasticsearch {
    hosts => "x.x.x.x"
    index => "winlog-*"
    #query => "*"
    size => 5000
    #scroll => "5m"
    docinfo => true
  }
}

output {
  elasticsearch {
    #index => "copy-of-prod.%{[@metadata][_index]}"
    index => "test"
    document_id => "%{[@metadata][_id]}"
  }
}

```

looking at the cluster during this code is running i see the "docs.count" maxed at 5000 and the "docs.deleted" change

```
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open test RNUXxuoIQrWKbjbsDTqZUw 5 1 5000 993 17.3mb 17.3mb

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open test RNUXxuoIQrWKbjbsDTqZUw 5 1 5000 1037 17.3mb 17.3mb

```

I know i can use \_reindex & elasticdump ... (looking into those), but wanted to understand what my mistake is

all elk sets are 5.4.0 one on centos7 other on RH7

Any help is much appreciated

thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 24, 2017, 6:00am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-problem/86847/2 "2017-05-24T06:00:56Z")

</div>

This works - [https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06](https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06)

Otherwise what version are you on?

---

<div class="post-metadata">

**Author:** ![orhiee](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@orhiee](https://discuss.elastic.co/u/orhiee)\
**Post date:** [May 24, 2017, 10:41am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-problem/86847/3 "2017-05-24T10:41:15Z")

</div>

Hi Mark,

when i directly copy the config you mentioned i get the foloowing:

```
11:24:02.311 [LogStash::Runner] ERROR logstash.inputs.elasticsearch - Unknown setting 'port' for elasticsearch
11:24:02.311 [LogStash::Runner] ERROR logstash.inputs.elasticsearch - Unknown setting 'scan' for elasticsearch
11:24:02.320 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:130:in `config_init'", "/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:62:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:98:in `plugin'", "(eval):8:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:63:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:145:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:274:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

```

and the scan and port are not mentioned in the latest doc ☹  
[https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html)

after commenting out the scan & port still got the same problem, only a 1000 records are in

`yellow open test l8GQd1r4RuOEmYNpjpKHlQ 5 1 1000 46 2mb 2mb`

the versions are 5.4.0 on both

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 24, 2017, 10:44am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-problem/86847/4 "2017-05-24T10:44:16Z")

</div>

> [@orhiee](#):
>
> and the scan and port are not mentioned in the latest doc ☹

Ahh, it may need updating then!

How many docs in the source index?

---

<div class="post-metadata">

**Author:** ![orhiee](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@orhiee](https://discuss.elastic.co/u/orhiee)\
**Post date:** [May 24, 2017, 12:39pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-problem/86847/5 "2017-05-24T12:39:22Z")

</div>

> [@warkolm](#):
>
> How many docs in the source index?

I create an index daily, about 40ish and 15million in total, indices goes from couple of k to 1.5 mil

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 21, 2017, 12:39pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-problem/86847/6 "2017-06-21T12:39:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.