Set timestamp from message

[Andrey_RF]

I have a pipeline to receive logs from different programs with different timestamp formats.

    input {
      beats {
        port => 5000
      }
    }

    filter {
      date {
        match => ["message", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
      }
    }


    output {
      elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "logstash-%{[host][hostname]}"
      }
    }

It puts log in elasticsearch but logs in ES have @timestamp equal to the time when they were saved in US, but not the creation date.

I have some logs examples:

[2020-08-28 14:00:02,940: ERROR/MainProcess] consumer: Cannot connect to ...

2021-04-01 15:06:49 =SUPERVISOR REPORT====
	Some other text

[stephenb (Stephen Brown)]

Hi @Andrey_RF Welcome to the community.

I suspect you need to add the timezone setting as part of your date filter, otherwise it is making an assumption.

Also please keep in mind all dates in elasticsearch are stored as UTC. If you look at them in Kibana Apps they will be displayed in your local timezone, if you display them directly via curl / API they will show in UTC

[Andrey_RF]

Hello. Thanks

I added timezone like this

filter {
  date {
    match => ["message", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
    timezone => "Europe/Moscow"
  }
}

and it doesn't help.

Yeah, I keep in my mind that ES are stored as UTC. But I had logs for 2020 and they have timestamp like April 1st 2021.

I can show you full log from ES if that help you.

[stephenb (Stephen Brown)]

I think I missed the obvious.

The code below will not extract the time out of your message field, you will need to parse the message first with a grok filter and then pass the field with the timestamp into it the date filter. So what is mostly likely happening that date filter is completely failing there is probably a tag in the document in elastic something like _dateparsefailure as so since it is failing it is just inserting the "now" time.

This wont work... because you are passing in a whole message not just the timestamp field

filter {
  date {
    match => ["message", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
    timezone => "Europe/Moscow"
  }
}

So your filter should look something like

filter {
  grok {
      match => {
      "message" => [
        # parse the message <!---- THESE ARE JUST EXAMPLES there can me multiple
      "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} \[%{DATA:class}\]:%{GREEDYDATA:message}",
      "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} \[%{DATA:class}\]:%{GREEDYDATA:message}"
     ]
    }
  }

  # Now you can parse the timestamp
 date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
    timezone => "Europe/Moscow"
  }
}

There are some good examples Here also lots of good articles on line

[Andrey_RF]

@stephenb

HESE ARE JUST EXAMPLES there can me multiple

Thanks for your quick reply. I realized my mistake and now it's working.