# Elasticsearch input not returning all data and Logstash closing

**URL:** <https://discuss.elastic.co/t/elasticsearch-input-not-returning-all-data-and-logstash-closing/220273>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [February 20, 2020, 10:17pm UTC](https://discuss.elastic.co/t/elasticsearch-input-not-returning-all-data-and-logstash-closing/220273 "2020-02-20T22:17:44Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![bradley.bristowstagg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley.bristowstagg/32/9954_2.png) [@bradley.bristowstagg](https://discuss.elastic.co/u/bradley.bristowstagg)\
**Post date:** [February 20, 2020, 10:17pm UTC](https://discuss.elastic.co/t/elasticsearch-input-not-returning-all-data-and-logstash-closing/220273/1 "2020-02-20T22:17:45Z")

</div>

Hey all,  
I'm using the current Logstash docker and am trying to pull all of the main "message" field from some existing indexes so that I can re-enrich them and move to a new index. Only problem is that logstash is closing without processing all of the data.

```
input {
  # Read all documents from Elasticsearch matching the given query
  elasticsearch {
    hosts => "10.1.1.40"
    index => 'smdr-*'
    query => '{ "query": { "match_all": {} }, "_source": ["message"]}'
    scroll => "15m"
    size => 10000
    user => "logstash_smdr"
    password => " *********"
  }
}#input

```

If I have a look at the original index it shows a document count of 116331, but the new index only has 1/10 that number of documents because Logstash is closing. Now I'm going to assume here that I've bollocksed something with the es input settings here and LS is closing because there's "nothing more to do".

From reading the docs setting a scroll should keep my query alive for the set period and initially I thought that might be the issue as I was not setting that. However even after setting it to 15m (see above) LS is still closing.

I also set the size to 10,000 and the latest run only pulled in 3750 records.

Help me elasticsearch community, you're my only hope....

....unless I stumble upon the solution myself 😃 🤣

---

<div class="post-metadata">

**Author:** ![bradley.bristowstagg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley.bristowstagg/32/9954_2.png) [@bradley.bristowstagg](https://discuss.elastic.co/u/bradley.bristowstagg)\
**Post date:** [February 20, 2020, 10:39pm UTC](https://discuss.elastic.co/t/elasticsearch-input-not-returning-all-data-and-logstash-closing/220273/2 "2020-02-20T22:39:07Z")

</div>

I set the log output to debug and can see that the input plugin is stopping for some reason:

```
[2020-02-19T02:34:37,952][DEBUG][logstash.javapipeline][pull-smdr] Input plugins stopped! Will shutdown filter/output workers. {:pipeline_id=>"pull-smdr", :thread=>"#<Thread:0x41e347b7 run>"}
[2020-02-19T02:34:37,975][DEBUG][logstash.javapipeline][pull-smdr] Shutdown waiting for worker thread {:pipeline_id=>"pull-smdr", :thread=>"#<Thread:0x3aeab159 run>"}

```

So what am I missing, why is the input plugin stopping when there's lots more data to pull?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 19, 2020, 10:39pm UTC](https://discuss.elastic.co/t/elasticsearch-input-not-returning-all-data-and-logstash-closing/220273/3 "2020-03-19T22:39:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.