# How to maintain the order of logs

**URL:** https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397
**Category:** Logstash
**Created:** [June 21, 2016, 4:28am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397 "2016-06-21T04:28:20Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![mattgolding](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@mattgolding](https://discuss.elastic.co/u/mattgolding)
#### Post date: [June 21, 2016, 4:28am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/1 "2016-06-21T04:28:20Z")

</div>

Hi I have a script which is bringing in log files and I am pushing them into Elasticsearch via Logstash and it all appears fine. However when searching over these logs I am finding it difficult to keep them in the order in which they appeared in the log file. I thought about adding the line of the log but looks like I cant access that in Logstash.

I do have the timesamp and it looks like this (see below) but as you can see alot of them have the exact time down to the MS

> 2016-06-14 13:53:42.5851 I 12 INFO  
> 2016-06-14 13:53:42.5851 I 12 INFO  
> 2016-06-14 13:53:42.5851 I 12 INFO  
> 2016-06-14 13:53:42.5851 I 12 INFO  
> 2016-06-14 13:53:42.7403 I 5 INFO  
> 2016-06-14 13:53:42.7403 I 5 INFO  
> 2016-06-14 13:53:42.7403 I 5 INFO  
> 2016-06-14 13:53:42.7564 I 5 INFO

---

<div class="post-metadata">

### Author: ![n1k](https://avatars.discourse-cdn.com/v4/letter/n/bbe5ce/32.png) [@n1k](https://discuss.elastic.co/u/n1k)
#### Post date: [June 23, 2016, 8:28am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/2 "2016-06-23T08:28:21Z")

</div>

I am also facing similar issue, are there any solutions or workaround?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 23, 2016, 8:37am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/3 "2016-06-23T08:37:12Z")

</div>

If they have exactly the same timestamp, you may need to include a line number or offset in the file when you index the data and include this when sorting.

---

<div class="post-metadata">

### Author: ![n1k](https://avatars.discourse-cdn.com/v4/letter/n/bbe5ce/32.png) [@n1k](https://discuss.elastic.co/u/n1k)
#### Post date: [June 23, 2016, 8:42am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/4 "2016-06-23T08:42:16Z")

</div>

Thanks Christian.  
It works

---

<div class="post-metadata">

### Author: ![mattgolding](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@mattgolding](https://discuss.elastic.co/u/mattgolding)
#### Post date: [June 23, 2016, 11:30am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/5 "2016-06-23T11:30:28Z")

</div>

N1k how did you get it to work? How did you get the offset? Is it in logstash or after?

---

<div class="post-metadata">

### Author: ![n1k](https://avatars.discourse-cdn.com/v4/letter/n/bbe5ce/32.png) [@n1k](https://discuss.elastic.co/u/n1k)
#### Post date: [June 23, 2016, 11:48am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/6 "2016-06-23T11:48:16Z")

</div>

I got offset in logstash after filtering.  
But one thing I am concerned about is when source log file gets rotated then again offset will start from 0, and how can we recollect the order. Need to check on this

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 1, 2016, 6:01pm UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/7 "2016-07-01T18:01:52Z")

</div>

> But one thing I am concerned about is when source log file gets rotated then again offset will start from 0, and how can we recollect the order. Need to check on this

Use the timestamp as primary sort key and the file offset as secondary?

---

<div class="post-metadata">

### Author: ![mattgolding](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@mattgolding](https://discuss.elastic.co/u/mattgolding)
#### Post date: [July 4, 2016, 11:15pm UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/8 "2016-07-04T23:15:19Z")

</div>

That duel sorting will work for me. Does anyone have an example of using the file offset as part of the logstash script?

---

<div class="post-metadata">

### Author: ![Mayank\_Agrawal](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)
#### Post date: [April 15, 2017, 6:36pm UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/9 "2017-04-15T18:36:20Z")

</div>

How did you get offset in logstash (in each event)?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:27am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-of-logs/53397/10 "2017-07-06T04:27:01Z")

</div>


