Logstash "elasticsearch input" problem

Elastic Stack Logstash
1

hello,

Couldnt find the answer on here or google, so here i am asking....

i want to use logstash to transfer data from one elasticsearch to another, but i cant move all the docs can only move the number spesified in the "size" parameter:

input {
 # Read all documents from Elasticsearch matching the given query
  elasticsearch {
    hosts => "x.x.x.x"
    index => "winlog-*"
    #query => "*"
    size => 5000
    #scroll => "5m"
    docinfo => true
  }
}

output {
  elasticsearch {
    #index => "copy-of-prod.%{[@metadata][_index]}"
    index => "test"
    document_id => "%{[@metadata][_id]}"
  }
}

looking at the cluster during this code is running i see the "docs.count" maxed at 5000 and the "docs.deleted" change

health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test    RNUXxuoIQrWKbjbsDTqZUw   5   1       5000          993     17.3mb         17.3mb

health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test    RNUXxuoIQrWKbjbsDTqZUw   5   1       5000         1037     17.3mb         17.3mb

I know i can use _reindex & elasticdump ... (looking into those), but wanted to understand what my mistake is

all elk sets are 5.4.0 one on centos7 other on RH7

Any help is much appreciated

thanks

2

This works - https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06

Otherwise what version are you on?

3

Hi Mark,

when i directly copy the config you mentioned i get the foloowing:

11:24:02.311 [LogStash::Runner] ERROR logstash.inputs.elasticsearch - Unknown setting 'port' for elasticsearch
11:24:02.311 [LogStash::Runner] ERROR logstash.inputs.elasticsearch - Unknown setting 'scan' for elasticsearch
11:24:02.320 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:130:in `config_init'", "/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:62:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:98:in `plugin'", "(eval):8:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:63:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:145:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:274:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

and the scan and port are not mentioned in the latest doc :frowning:
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

after commenting out the scan & port still got the same problem, only a 1000 records are in

yellow open test l8GQd1r4RuOEmYNpjpKHlQ 5 1 1000 46 2mb 2mb

the versions are 5.4.0 on both

4

Ahh, it may need updating then!

How many docs in the source index?

5

I create an index daily, about 40ish and 15million in total, indices goes from couple of k to 1.5 mil

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.