# How to Restrict Logstash to process new records only after previous record is completely processed **URL:** \ **Category:** Logstash\ **Created:** [August 28, 2018, 10:29am UTC](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320 "2018-08-28T10:29:24Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Arjun](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Arjun](https://discuss.elastic.co/u/Arjun)\ **Post date:** [August 28, 2018, 10:29am UTC](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320/1 "2018-08-28T10:29:24Z")
Hi Team, Whenever new record enters Logstash it should wait until previous record is completely processed only then new record should be processed by Logstash. Please suggest me how to configure this in Logstash. Thanks Arjun ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 28, 2018, 6:47pm UTC](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320/2 "2018-08-28T18:47:44Z")
Try setting the batch size and pipeline workers to 1 (one). See [https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html](https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html). This is a weird requirement. You might get more relevant help if you explain the background. ---
**Author:** ![Arjun](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Arjun](https://discuss.elastic.co/u/Arjun)\ **Post date:** [August 30, 2018, 10:11am UTC](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320/3 "2018-08-30T10:11:37Z")
My requirement is - Each time I get a record to logstash for processing I need to query elastic search for related records and copy certain fields from it to current record. Whenever I load a file, related records exists within the file as well because of which I need the records to process one at a time or else I want the related events/records to wait till the current processing related event is completely processed and put to elastic search. Please let me know if you need some more background so that I can explain. Thanks Arjun ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 30, 2018, 11:08am UTC](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320/4 "2018-08-30T11:08:19Z")
How are you locating the related records, via a query in an elasticsearch filter? ---
**Author:** ![Arjun](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Arjun](https://discuss.elastic.co/u/Arjun)\ **Post date:** [August 30, 2018, 11:34am UTC](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320/5 "2018-08-30T11:34:13Z")
Yes in Logstash Filter Plugin I am using Elasticsearch plugin and inside that I am using the query template pointing to the file which has elastic search query. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 30, 2018, 12:01pm UTC](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320/6 "2018-08-30T12:01:18Z")
That'll _never_ work reliably since queries don't have strong consistency guarantees. What that means in practice is that newly added documents don't become searchable for a few seconds (depending on the configuration). It's possible to force them to be available immediately by requesting a refresh operation, but that's too costly for each document. I think you need to consider another option, e.g. preprocessing the input elsewhere or writing a custom Logstash plugin that both updates a cache somewhere with the current document but also looks up related documents in the same cache. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 27, 2018, 12:01pm UTC](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320/7 "2018-09-27T12:01:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.