# How to maintain the order in which records are read

**URL:** <https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015>\
**Category:** Logstash\
**Created:** [June 27, 2017, 6:53pm UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015 "2017-06-27T18:53:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tejas\_Ghutukade](https://avatars.discourse-cdn.com/v4/letter/t/ba9def/32.png) [@Tejas\_Ghutukade](https://discuss.elastic.co/u/Tejas_Ghutukade)\
**Post date:** [June 27, 2017, 6:53pm UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015/1 "2017-06-27T18:53:49Z")

</div>

M trying to read a csv file which is sorted by time-stamp.

m trying to add a field with incremental value as per the ascending order of time-stamp

But the incremental value is getting assigned in random order to the records.  
how to assign the values in the order of the time-stamp.

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 27, 2017, 7:59pm UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015/2 "2017-06-27T19:59:55Z")

</div>

What do your filters look like? A first step would be to restrict the number of pipeline workers to 1 (one).

---

<div class="post-metadata">

**Author:** ![Tejas\_Ghutukade](https://avatars.discourse-cdn.com/v4/letter/t/ba9def/32.png) [@Tejas\_Ghutukade](https://discuss.elastic.co/u/Tejas_Ghutukade)\
**Post date:** [June 27, 2017, 8:06pm UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015/3 "2017-06-27T20:06:13Z")

</div>

Here's my filter.

```
filter {
csv{
	separator => ","
	columns => ["loglevel","taskid","logger","label","duration","ttime","ptime"]
	convert => { 
		"duration" => "integer"
		"ptime" => "integer"
		"ttime" => "date"			
	}

aggregate {
	task_id => "%{taskid}"
	code => "
       map['test'] ||= 0;
       event.set('pp',map['test']);
		map['test'] += 1;
     "
	push_map_as_event_on_timeout => true
	timeout_task_id_field => "taskid"
	timeout => 30
	timeout_tags => ['_aggregatetimeout']
	timeout_code => ""
  }
 }

```

how can i restrict the number of pipeline workers?

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 27, 2017, 8:15pm UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015/4 "2017-06-27T20:15:56Z")

</div>

> how can i restrict the number of pipeline workers?

> **[logstash.yml | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html)**

---

<div class="post-metadata">

**Author:** ![Tejas\_Ghutukade](https://avatars.discourse-cdn.com/v4/letter/t/ba9def/32.png) [@Tejas\_Ghutukade](https://discuss.elastic.co/u/Tejas_Ghutukade)\
**Post date:** [June 27, 2017, 8:28pm UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015/5 "2017-06-27T20:28:38Z")

</div>

Thanks it worked..  
i changes the pipeline workers to 1.  
will it affect for large amount of data(~600k records)?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 28, 2017, 6:06am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015/6 "2017-06-28T06:06:45Z")

</div>

If you don't allow filters to process events in parallel the performance will of course suffer.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 26, 2017, 6:06am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015/7 "2017-07-26T06:06:45Z")

</div>

Closed on July 26, 2017, 6:06am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.