Logstash "elasticsearch input" problem


(Orhan) #1

hello,

Couldnt find the answer on here or google, so here i am asking....

i want to use logstash to transfer data from one elasticsearch to another, but i cant move all the docs can only move the number spesified in the "size" parameter:

input {
 # Read all documents from Elasticsearch matching the given query
  elasticsearch {
    hosts => "x.x.x.x"
    index => "winlog-*"
    #query => "*"
    size => 5000
    #scroll => "5m"
    docinfo => true
  }
}

output {
  elasticsearch {
    #index => "copy-of-prod.%{[@metadata][_index]}"
    index => "test"
    document_id => "%{[@metadata][_id]}"
  }
}

looking at the cluster during this code is running i see the "docs.count" maxed at 5000 and the "docs.deleted" change

health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test    RNUXxuoIQrWKbjbsDTqZUw   5   1       5000          993     17.3mb         17.3mb

health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test    RNUXxuoIQrWKbjbsDTqZUw   5   1       5000         1037     17.3mb         17.3mb

I know i can use _reindex & elasticdump ... (looking into those), but wanted to understand what my mistake is

all elk sets are 5.4.0 one on centos7 other on RH7

Any help is much appreciated

thanks


(Mark Walkom) #2

This works - https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06

Otherwise what version are you on?


(Orhan) #3

Hi Mark,

when i directly copy the config you mentioned i get the foloowing:

11:24:02.311 [LogStash::Runner] ERROR logstash.inputs.elasticsearch - Unknown setting 'port' for elasticsearch
11:24:02.311 [LogStash::Runner] ERROR logstash.inputs.elasticsearch - Unknown setting 'scan' for elasticsearch
11:24:02.320 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:130:in `config_init'", "/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:62:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:98:in `plugin'", "(eval):8:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:63:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:145:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:274:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

and the scan and port are not mentioned in the latest doc :frowning:
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

after commenting out the scan & port still got the same problem, only a 1000 records are in

yellow open test l8GQd1r4RuOEmYNpjpKHlQ 5 1 1000 46 2mb 2mb

the versions are 5.4.0 on both


(Mark Walkom) #4

Ahh, it may need updating then!

How many docs in the source index?


(Orhan) #5

I create an index daily, about 40ish and 15million in total, indices goes from couple of k to 1.5 mil


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.