Log messages with same index timestamps are not in the order as in the actual log file

[Aj_t]

I have a filebeat > logstash > elastic set up I am using to process multi-line logs. I can see the logs in kibana, but they are not in the right order as the actual log.

Here's my filebeat.yml file:

filebeat.inputs:
- type: log
  enabled: true

  paths:
    - /var/log/some.log

  ### Multiline options
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["xx.xx.xx.xx:5044"]

I have also set the pipeline.workers to 1 (-w 1) in the logstash.yml , so that shouldn't be an issue.

From what I've read, there are some suggestions about setting the offset. I am not sure where or how to do this to get the output in the right order. Would this be set up in filebeat or logstash?

Here's the sample output in kibana:

As you can see, they were all indexed at the same time (Time), but I want the output to be sorted by time field.

[Jenni]

Time is just a text, isn't it? Then you should turn this into an actual Timestamp, e.g.:

mutate {
  replace => { "time" => "%{+yyyy.MM.dd} %{time}" }
}
date {
  match => ["time", "yyyy.MM.dd HH:mm:ss,SSS"]
  target => "time"
}

Then you can select time as your timestamp field for your index pattern in Kibana.

[Aj_t]

Yes, time is just text. So I added your code to my logstash filter after my grok pattern. But I still have the same issue - logs are not in the right order.

Not sure what's going on

[Jenni]

You'll have to select time as the field to be sorted by in Kibana. Otherwise that won't change.
(Edit: Or maybe you want to overwrite Time with the Timestamp that I was trying to create, if there isn't any specific reason for you to save the timestamp of the import in addition to the log timestamp)

---

Btw.: If date belongs to time we probably should have used that for the parsed string? The config I had posted earlier sets time to the day of @timestamp, so it's the 21st in your latest screenshot, not the 20th.

mutate {
  replace => { "time" => "%{date} %{time}" }
}
date {
  match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]
  target => "time"
}

[Aj_t]

You are right - the date should be 7/20 as that's the date the logs have - fixed now with your suggestion.

I think the time is now in Zero timezone (?). I would like to keep it in the local timezone just like what I have in the logs. How would I do that?

I guess I am struggling a bit to sort by time in kibana. Here's my logstash output setup:

output {

  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Do I have to change something in the index line to sort by time rather than timestamp? Thanks so much!

[Jenni]

I can't check that in Kibana at the moment. But I think that there is a sort button if you hover your cursor over the column headings? And I think the default sort field of the Discover view is the field you selected as the time field when you created the index pattern.

Timestamps are saved internally as UTC, but should be displayed in your local timezone in Kibana. I think that the problem is that time is still not a Timestamp, but a string because you didn't reindex your data in a new index. The index you are currently indexing your documents into has already mapped time as a text data type and that can not be changed afterwards.

[Jenni]

Now it's morning for me and I had a chance to have a look. There is a sort button, but it's not there for fields with the data type text because you can't use these for sorting. So your time is probably text, not a keyword or date as these would be sortable.
So if you don't need your import time in your index, I would suggest overwriting @timestamp with the time of the log.

mutate {
  add_field => { "[@metadata][tmp_ts_string]" => "%{date} %{time}" }
}
date {
  match => ["[@metadata][tmp_ts_string]", "yyyy-MM-dd HH:mm:ss,SSS"]
}

If you do need it, you'll have to figure out why time isn't mapped as a date even after creating a new index. (Make sure it isn't by checking the mapping of the index first and refreshing the field list for your index pattern in Kibana if the ES mapping and Kibana contradict each other.) Maybe you'll have to set the mapping for your index manually or set up an index template to make sure it becomes a date. But if it is a date at its first appearance, the automatic mapping should actually get it right.

[Aj_t]

It turned out to be a mapping related issue.

With this suggestion of yours in the logstash pipeline config

mutate {
  replace => { "time" => "%{date} %{time}" }
}
date {
  match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]
  target => "time"
}

I was able to get most of it fixed.

Prior to this, I was not creating an index or mapping anything manually and I was just defining which index the logs should go into in the logstash config output. Without the above piece of code, the mapping wasn't happening the right way and time was still being treated as text.

Yes, with this, I am loosing the time the logs were injected into elastic, but I do not think I am going to need it. The time in the logs is what's important to me.

However, now I have some issues with the date field. Sure, I wouldn't need it because I have the time, but can't figure out why it says July 20th and 19:00:00.000 for every single line.

[Jenni]

You mapped the date as a date. So ES turned this date into 2020-07-21T00:00:00.000Z and I guess you are UTC-5? Kibana displays this in your local time zone.

[Aj_t]

Yes, that makes sense. I tried to drop the date field in logstash via remove_field at the very last after using it to derive the time field, and that seems to be working. I might also look into replacing @timestamp with the time of the log.

I still have bunch of other stuff to figure out - will start new threads for those question. Thanks for your help so far.