# Log messages with same index timestamps are not in the order as in the actual log file
**URL:** https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000
**Category:** Logstash
**Created:** [July 21, 2020, 10:13am UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000 "2020-07-21T10:13:55Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Aj\_t](https://discuss.elastic.co/u/Aj_t)
#### Post date: [July 21, 2020, 10:13am UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/1 "2020-07-21T10:13:55Z")
I have a **filebeat \> logstash \> elastic** set up I am using to process multi-line logs. I can see the logs in kibana, but they are not in the right order as the actual log.
Here's my filebeat.yml file:
```auto
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/some.log
### Multiline options
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["xx.xx.xx.xx:5044"]
```
I have also set the **pipeline.workers** to 1 (-w 1) in the logstash.yml , so that shouldn't be an issue.
From what I've read, there are some suggestions about setting the **offset**. I am not sure where or how to do this to get the output in the right order. Would this be set up in filebeat or logstash?
Here's the sample output in kibana:
As you can see, they were all indexed at the same time (`Time`), but I want the output to be sorted by **`time`** field.
---
### Author:  [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [July 21, 2020, 11:09am UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/2 "2020-07-21T11:09:28Z")
Time is just a text, isn't it? Then you should turn this into an actual Timestamp, e.g.:
```auto
mutate {
replace => { "time" => "%{+yyyy.MM.dd} %{time}" }
}
date {
match => ["time", "yyyy.MM.dd HH:mm:ss,SSS"]
target => "time"
}
```
Then you can select `time` as your timestamp field for your index pattern in Kibana.
---
### Author:  [@Aj\_t](https://discuss.elastic.co/u/Aj_t)
#### Post date: [July 21, 2020, 8:52pm UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/3 "2020-07-21T20:52:28Z")
Yes, `time` is just text. So I added your code to my logstash filter after my grok pattern. But I still have the same issue - logs are not in the right order.
Not sure what's going on
---
### Author:  [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [July 21, 2020, 9:02pm UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/4 "2020-07-21T21:02:16Z")
You'll have to select `time` as the field to be sorted by in Kibana. Otherwise that won't change.
(Edit: Or maybe you want to overwrite `Time` with the Timestamp that I was trying to create, if there isn't any specific reason for you to save the timestamp of the import in addition to the log timestamp)
* * *
Btw.: If `date` belongs to `time` we probably should have used that for the parsed string? The config I had posted earlier sets time to the day of @timestamp, so it's the 21st in your latest screenshot, not the 20th.
```auto
mutate {
replace => { "time" => "%{date} %{time}" }
}
date {
match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]
target => "time"
}
```
---
### Author:  [@Aj\_t](https://discuss.elastic.co/u/Aj_t)
#### Post date: [July 21, 2020, 9:19pm UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/5 "2020-07-21T21:19:48Z")
You are right - the `date` should be 7/20 as that's the date the logs have - fixed now with your suggestion.
I think the `time` is now in Zero timezone (?). I would like to keep it in the local timezone just like what I have in the logs. How would I do that?
I guess I am struggling a bit to sort by `time` in kibana. Here's my logstash output setup:
```auto
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
```
Do I have to change something in the index line to sort by `time` rather than `timestamp`? Thanks so much!
---
### Author:  [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [July 21, 2020, 9:28pm UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/6 "2020-07-21T21:28:09Z")
I can't check that in Kibana at the moment. But I think that there is a sort button if you hover your cursor over the column headings? And I think the default sort field of the Discover view is the field you selected as the time field when you created the index pattern.
Timestamps are saved internally as UTC, but should be displayed in your local timezone in Kibana. I think that the problem is that `time` is still not a Timestamp, but a string because you didn't reindex your data in a new index. The index you are currently indexing your documents into has already mapped `time` as a text data type and that can not be changed afterwards.
---
### Author:  [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [July 22, 2020, 7:34am UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/7 "2020-07-22T07:34:45Z")
Now it's morning for me and I had a chance to have a look. There is a sort button, but it's not there for fields with the data type `text` because you can't use these for sorting. So your time is probably `text`, not a `keyword` or `date` as these would be sortable.
So if you don't need your import time in your index, I would suggest overwriting `@timestamp` with the time of the log.
```auto
mutate {
add_field => { "[@metadata][tmp_ts_string]" => "%{date} %{time}" }
}
date {
match => ["[@metadata][tmp_ts_string]", "yyyy-MM-dd HH:mm:ss,SSS"]
}
```
If you do need it, you'll have to figure out why `time` isn't mapped as a `date` even after creating a new index. (Make sure it isn't by checking the mapping of the index first and refreshing the field list for your index pattern in Kibana if the ES mapping and Kibana contradict each other.) Maybe you'll have to set the mapping for your index manually or set up an index template to make sure it becomes a `date`. But if it is a date at its first appearance, the automatic mapping should actually get it right.
---
### Author:  [@Aj\_t](https://discuss.elastic.co/u/Aj_t)
#### Post date: [July 22, 2020, 8:04am UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/8 "2020-07-22T08:04:14Z")
It turned out to be a mapping related issue.
With this suggestion of yours in the logstash pipeline config
```auto
mutate {
replace => { "time" => "%{date} %{time}" }
}
date {
match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]
target => "time"
}
```
I was able to get most of it fixed.
Prior to this, I was not creating an index or mapping anything manually and I was just defining which index the logs should go into in the logstash config output. Without the above piece of code, the mapping wasn't happening the right way and `time` was still being treated as text.
Yes, with this, I am loosing the `time` the logs were injected into elastic, but I do not think I am going to need it. The `time` in the logs is what's important to me.
However, now I have some issues with the `date` field. Sure, I wouldn't need it because I have the `time`, but can't figure out why it says July 20th and 19:00:00.000 for every single line.
---
### Author:  [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [July 22, 2020, 8:13am UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/9 "2020-07-22T08:13:31Z")
You mapped the date as a `date`. So ES turned this date into `2020-07-21T00:00:00.000Z` and I guess you are UTC-5? Kibana displays this in your local time zone.
---
### Author:  [@Aj\_t](https://discuss.elastic.co/u/Aj_t)
#### Post date: [July 22, 2020, 9:12am UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/10 "2020-07-22T09:12:27Z")
Yes, that makes sense. I tried to drop the `date` field in logstash via `remove_field` at the very last after using it to derive the `time` field, and that seems to be working. I might also look into replacing `@timestamp` with the time of the log.
I still have bunch of other stuff to figure out - will start new threads for those question. Thanks for your help so far.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 19, 2020, 9:12am UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000/11 "2020-08-19T09:12:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.