# How to control the filebeat read file sequence **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 10, 2016, 3:55am UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973 "2016-03-10T03:55:04Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![szcountryboy](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@szcountryboy](https://discuss.elastic.co/u/szcountryboy)\ **Post date:** [March 10, 2016, 3:55am UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973/1 "2016-03-10T03:55:04Z")
The was log is stored in the order, but the order of the filebeat is changed, as shown below: Time Message March 10th 2016, 11:43:41.721 [16-2-3 0:01:04:356 CST] 00000026 SystemErr R log4j:WARN Please initia... March 10th 2016, 11:43:41.721 [16-2-4 17:33:58:032 CST] 00000038 SystemErr R log4j:WARN No appende... March 10th 2016, 11:43:41.721 [16-2-1 3:23:18:644 CST] 00000025 SystemErr R at com.ibm.ws.webcontai Is it because of the filebeat issue, or the ES display? Click message sort, also did not sort by time Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 10, 2016, 6:59am UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973/2 "2016-03-10T06:59:13Z")
ES only stores millisecond resolution and since all three messages have the same timestamp it can't sort them any better. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [March 18, 2016, 1:31pm UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973/3 "2016-03-18T13:31:55Z")
Can you check the 'offset' of indexed documents in elasticsearch? ---
**Author:** ![Alti](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@Alti](https://discuss.elastic.co/u/Alti)\ **Post date:** [April 14, 2016, 12:29pm UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973/4 "2016-04-14T12:29:43Z")
Hi All, We have faced with the same issue, looks like it happens because filebeat sends bunch of logs as one bulk message to logstash, as result all of source messages will be with the same @timestamp and when you query them in Kibana they returns in random order because of sorting by @timestamp... which is equal for set of messages. Is there any options to avoid such behaviour? I would be happy if beats could send messages one by one as their appear in file without merging them in to bulk inserts. We've tried changing scan\_frequency and bulk\_max\_size but it doesn't really help. Any thoughts on this? Regards. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 14, 2016, 12:36pm UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973/5 "2016-04-14T12:36:36Z")
The `@timestamp` value shouldn't be the one added by Logstash. It should be picked up from the log message itself. ---
**Author:** ![szcountryboy](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@szcountryboy](https://discuss.elastic.co/u/szcountryboy)\ **Post date:** [April 15, 2016, 1:46am UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973/6 "2016-04-15T01:46:04Z")
Thanks your reply. Background of the problem is that I want to monitor tomcat log information. You know, there may be anomalies in the log message inside, and exception occurs when there is a stack of information. From the user point of view, it should be part of a message (as a whole), and if each message is divided randomly displayed for the user to read may be very inconvenient. I use the following method to solve: 1. I log output format has been adjusted, each line of the log header fixed identity, such as: 2016 and so on. 2. Then filebeat profile use regular expressions inside the merge process Thank you. ------------------ 原始邮件 ------------------ 发件人: "Magnus Bäck";[noreply@elastic.co](mailto:noreply@elastic.co); 发送时间: 2016年4月14日(星期四) 晚上8:46 收件人: "szcountryboy"[15932551@qq.com](mailto:15932551@qq.com); 主题: [Beats/Filebeat] How to control the filebeat read file sequence ``` magnusbaeck Magnus Bäck Logstash Plugins Community Maintainer April 14 ``` The @timestamp value shouldn't be the one added by Logstash. It should be picked up from the log message itself. Visit Topic or reply to this email to respond In Reply To Alti Aleksey Timchenko April 14 ``` Hi All, We have faced with the same issue, looks like it happens because filebeat sends bunch of logs as one bulk message to logstash, as result all of source messages will be with the same @timestamp and when you query them in Kibana they returns in random order because of sorting by @timestamp.… ``` Visit Topic or reply to this email to respond To stop receiving notifications for this particular topic, click here. To unsubscribe from these emails, change your user preferences ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 15, 2016, 1:40pm UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973/7 "2016-04-15T13:40:02Z")
filebeat is assigning the timestamp when reading files. Normally one uses grok in logstash to parser the log messages and extract the timestamp from original log-messages. This gets you the correct order. When dealing with exceptions, the preferred solution is to use multiline support in filebeat to combine message + full stack trace into one event (so message + full stack trace) can be read/indexed as one entity. =\> Proper use of multiline prevents a message being divided randomly... ---
**Author:** ![Alti](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@Alti](https://discuss.elastic.co/u/Alti)\ **Post date:** [April 26, 2016, 3:54pm UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973/8 "2016-04-26T15:54:14Z")
Thank you for response, it was very useful. Previously we were using logstash-forwarder/nlog/nxlog and sorting by @timestamp generated by logstash and it's satisfied our needs. Now, to meet Beats Approach, we just changed our elasticsearch index templates, modified date type of field which represents original log entity time (written by log source application) from string to "Date" and in Kibana switched to this, new, "Time-field name" in index Patterns ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:52pm UTC](https://discuss.elastic.co/t/how-to-control-the-filebeat-read-file-sequence/43973/9 "2017-07-05T21:52:46Z")