Hey all,
I'm using the current Logstash docker and am trying to pull all of the main "message" field from some existing indexes so that I can re-enrich them and move to a new index. Only problem is that logstash is closing without processing all of the data.
input {
# Read all documents from Elasticsearch matching the given query
elasticsearch {
hosts => "10.1.1.40"
index => 'smdr-*'
query => '{ "query": { "match_all": {} }, "_source": ["message"]}'
scroll => "15m"
size => 10000
user => "logstash_smdr"
password => "*********"
}
}#input
If I have a look at the original index it shows a document count of 116331, but the new index only has 1/10 that number of documents because Logstash is closing. Now I'm going to assume here that I've bollocksed something with the es input settings here and LS is closing because there's "nothing more to do".
From reading the docs setting a scroll should keep my query alive for the set period and initially I thought that might be the issue as I was not setting that. However even after setting it to 15m (see above) LS is still closing.
I also set the size to 10,000 and the latest run only pulled in 3750 records.
Help me elasticsearch community, you're my only hope....
....unless I stumble upon the solution myself 
