# Auto\_flush\_interval with ordered pipeline

**URL:** <https://discuss.elastic.co/t/auto-flush-interval-with-ordered-pipeline/271924>\
**Category:** Logstash\
**Created:** [May 3, 2021, 4:07am UTC](https://discuss.elastic.co/t/auto-flush-interval-with-ordered-pipeline/271924 "2021-05-03T04:07:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![roua.B](https://avatars.discourse-cdn.com/v4/letter/r/91b2a8/32.png) [@roua.B](https://discuss.elastic.co/u/roua.B)\
**Post date:** [May 3, 2021, 4:07am UTC](https://discuss.elastic.co/t/auto-flush-interval-with-ordered-pipeline/271924/1 "2021-05-03T04:07:10Z")

</div>

Hello,  
So I was facing an issue with logstash not parsing the last line of my xml log file. I found that the answer would be to add `auto_flush_interval => 1` to my multiline codec. I would like my pipeline to be ordered so I set `pipeline.workers : 1` and by default `pipeline.ordered: true`. This works just fine when it's just one file. My problem is that I have to parse multiple files at once, and the order gets messed up with the `auto_flush_interval` enabled.

Is there any way around this issue?

Thank you 🙂

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 3, 2021, 2:47pm UTC](https://discuss.elastic.co/t/auto-flush-interval-with-ordered-pipeline/271924/2 "2021-05-03T14:47:59Z")

</div>

Are you saying that the files are not read in the order you expect? If so, the file input does not guarantee order, so this is normal.

---

<div class="post-metadata">

**Author:** ![roua.B](https://avatars.discourse-cdn.com/v4/letter/r/91b2a8/32.png) [@roua.B](https://discuss.elastic.co/u/roua.B)\
**Post date:** [May 3, 2021, 7:16pm UTC](https://discuss.elastic.co/t/auto-flush-interval-with-ordered-pipeline/271924/3 "2021-05-03T19:16:57Z")

</div>

The order in which the files are read doesn't matter, what matters is that the lines of each file are read in order. I would like for logstash to complete the whole file before passing to the next one. What actually happens is, it continues to parse the next file before reading the last line of the previous one. Is there a condition to put or some way to assert that it does not pass to the next file unless it reads the last line?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 3, 2021, 7:41pm UTC](https://discuss.elastic.co/t/auto-flush-interval-with-ordered-pipeline/271924/4 "2021-05-03T19:41:34Z")

</div>

In the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_description_11) it talks about "complete" reading as opposed to "stripped" reading where it interleaves chunks of files. The default options make it read 4611686018427387903 chunks of 32 KB from a file before it starts processing the next one. Effectively, it reads to EOF. I cannot think why a multiline codec would change that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 31, 2021, 7:42pm UTC](https://discuss.elastic.co/t/auto-flush-interval-with-ordered-pipeline/271924/5 "2021-05-31T19:42:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.