Elasticsearch input not returning all data and Logstash closing

Elastic Stack Logstash
1

Hey all,
I'm using the current Logstash docker and am trying to pull all of the main "message" field from some existing indexes so that I can re-enrich them and move to a new index. Only problem is that logstash is closing without processing all of the data.

input {
  # Read all documents from Elasticsearch matching the given query
  elasticsearch {
    hosts => "10.1.1.40"
    index => 'smdr-*'
    query => '{ "query": { "match_all": {} }, "_source": ["message"]}'
    scroll => "15m"
    size => 10000
    user => "logstash_smdr"
    password => "*********"
  }
}#input

If I have a look at the original index it shows a document count of 116331, but the new index only has 1/10 that number of documents because Logstash is closing. Now I'm going to assume here that I've bollocksed something with the es input settings here and LS is closing because there's "nothing more to do".

From reading the docs setting a scroll should keep my query alive for the set period and initially I thought that might be the issue as I was not setting that. However even after setting it to 15m (see above) LS is still closing.

I also set the size to 10,000 and the latest run only pulled in 3750 records.

Help me elasticsearch community, you're my only hope....

....unless I stumble upon the solution myself :smiley: :rofl:

2

I set the log output to debug and can see that the input plugin is stopping for some reason:

[2020-02-19T02:34:37,952][DEBUG][logstash.javapipeline    ][pull-smdr] Input plugins stopped! Will shutdown filter/output workers. {:pipeline_id=>"pull-smdr", :thread=>"#<Thread:0x41e347b7 run>"}
[2020-02-19T02:34:37,975][DEBUG][logstash.javapipeline    ][pull-smdr] Shutdown waiting for worker thread {:pipeline_id=>"pull-smdr", :thread=>"#<Thread:0x3aeab159 run>"}

So what am I missing, why is the input plugin stopping when there's lots more data to pull?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.