How to setup logstash workers or batch size

i have es cluster 6.1.3
i want migrate data to es 7.17.9
i used logstash 7.17.9 to make it
my index count has only 1200 doc

my logstash config file:

input {
  elasticsearch {
    hosts => ["10.251.0.11:39202","10.251.0.12:39202","10.251.0.13:39202","10.251.0.14:39202","10.251.0.15:39202"]
    index => "new_index_001"
    query => '{ "query": { "match_all": {} } }'
    docinfo => true
  }
}

filter {
}

output {
  elasticsearch {
    hosts => [["10.251.0.11:39201","10.251.0.12:39201","10.251.0.13:39201","10.251.0.14:39201","10.251.0.15:39201"]]
    index => "new_index_001_v7"
  }
}

startup cmd:
../bin/logstash -f es6_to_es7.conf -b 20000 -w 500

but when i was startup logstash, it's shutdown immediate and data only sync little

error:
[2023-07-07T23:18:12,280][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>500, "pipeline.batch.size"=>16000, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>8000000, "pipeline.sources"=>["/home/kafka/logstash-7.17.9/config/es6_to_es7.conf"], :thread=>"#<Thread:0x264ebb94 run>"}
[2023-07-07T23:18:12,283][WARN ][logstash.javapipeline    ][main] CAUTION: Recommended inflight events max exceeded! Logstash will run with up to 8000000 events in memory in your current configuration. If your message sizes are large this may cause instability with the default heap size. Please consider setting a non-standard heap size, changing the batch size (currently 16000), or changing the number of pipeline workers (currently 500) {:pipeline_id=>"main", :thread=>"#<Thread:0x264ebb94 run>"}
[2023-07-07T23:18:18,209][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>5.92}
[2023-07-07T23:18:18,778][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-07-07T23:18:18,904][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-07-07T23:18:22,021][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-07-07T23:18:22,150][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2023-07-07T23:18:22,227][INFO ][logstash.runner          ] Logstash shut down.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.