# Documents order in Elasticsearch after processing with filebeat/logstash **URL:** \ **Category:** Logstash\ **Created:** [August 5, 2017, 1:57pm UTC](https://discuss.elastic.co/t/documents-order-in-elasticsearch-after-processing-with-filebeat-logstash/95980 "2017-08-05T13:57:13Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![njkl](https://avatars.discourse-cdn.com/v4/letter/n/97f17d/32.png) [@njkl](https://discuss.elastic.co/u/njkl)\ **Post date:** [August 5, 2017, 1:57pm UTC](https://discuss.elastic.co/t/documents-order-in-elasticsearch-after-processing-with-filebeat-logstash/95980/1 "2017-08-05T13:57:13Z")
Hi, I have the following setup to gather data : filebeat -\> logstash -\> elastichsearch As input I have files with the following plain text format : 2017-07-25 10:36:07,988 21 User 1 Ligne 0 2017-07-25 10:36:07,988 21 User 1 Ligne 1 2017-07-25 10:36:07,988 21 none 1 Ligne 2 once the file is harvested by filebeat it goes thru logstash where I map @timestamp field. but when I query Elasticsearch I get different order than the one the source log file. (log2, log 0, log 1 instead of log 0, log 1, log 2 ) the timestamp field in not precise enough I will always get more than 1 doc for the same exact timestamp. How can I solve this issue ? I would like to be able to get document in elasticsearch in the same order as their are in the log file. Regards ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 5, 2017, 3:28pm UTC](https://discuss.elastic.co/t/documents-order-in-elasticsearch-after-processing-with-filebeat-logstash/95980/2 "2017-08-05T15:28:49Z")
Perhaps you can use the file offset as a secondary sort key? I believe Filebeat records it in the `offset` field. ---
**Author:** ![njkl](https://avatars.discourse-cdn.com/v4/letter/n/97f17d/32.png) [@njkl](https://discuss.elastic.co/u/njkl)\ **Post date:** [August 6, 2017, 12:08pm UTC](https://discuss.elastic.co/t/documents-order-in-elasticsearch-after-processing-with-filebeat-logstash/95980/3 "2017-08-06T12:08:40Z")
You are right, if I use both @timestamp and the offset. the result seems to be good. but I have one more question. Do you know how can I merge @timestamp and offset into a new field and use it as unique sorting criteria (I suppose its better when it comes to reindexing and performance) update : I tried this > add\_field =\> { "timestamp\_sort" =\> "%{@timestamp}+%{offset}" } But as result I got this : 😕 > "timestamp\_sort": " **2017-08-06T08:36:07.988Z+53**" ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 6, 2017, 6:49pm UTC](https://discuss.elastic.co/t/documents-order-in-elasticsearch-after-processing-with-filebeat-logstash/95980/4 "2017-08-06T18:49:51Z")
You'd have to use a ruby filter to do that, but I don't think just adding the timestamp expressed in milliseconds (if that's what you meant) with the file offset is a good idea since entries right after a rotation could very well sort prior to entries just before the rotation. You should figure out something else that doesn't have that problem. ---
**Author:** ![njkl](https://avatars.discourse-cdn.com/v4/letter/n/97f17d/32.png) [@njkl](https://discuss.elastic.co/u/njkl)\ **Post date:** [August 22, 2017, 5:10pm UTC](https://discuss.elastic.co/t/documents-order-in-elasticsearch-after-processing-with-filebeat-logstash/95980/5 "2017-08-22T17:10:31Z")
Hi Magnus, Thank you for your feedback. So far, I was able to figure out the following : - declare timestamp\_sort as date in the mapping of the index in elasticsearch > "timestamp\_sort" : { > "type" : "date", > "format": "yyyy-MM-dd HH:mm:ss,SSSSSSSSSSSSSSSS" > } - in logstash simply concatenate timestamp and the offset (not adding but **concatenate** ) > add\_field =\> { “timestamp\_sort” =\> “%{@timestamp}%{offset}” } when pulling data from ELK, the sorting will be based on the new field (asc) + file name (desc). This seems to be working, at least I didn't see any issue so far 😃 Regards ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 19, 2017, 5:10pm UTC](https://discuss.elastic.co/t/documents-order-in-elasticsearch-after-processing-with-filebeat-logstash/95980/6 "2017-09-19T17:10:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.