Replace @timestamp with actual timestamp from log file

Hi,
I am trying to index my mail-relay log files to Elastic search. All the log entries are being indexed into a field named message . @timestamp field shows the time the entry was indexed and not the timestamp from the log entry.

Below are the source log file lines sample:

2021-04-21 15:00:03 104.47.58.138 OutboundConnectionCommand SMTPSVC1 LWMAILVU1 - 25 BDAT - 10695+LAST 0 0 4 0 1969 SMTP - - - -
2021-04-21 15:00:03 104.47.58.138 OutboundConnectionResponse SMTPSVC1 LWMAILVU1 - 25 - - 250+2.1.5+Recipient+OK 0 0 22 0 1969 SMTP - - - -

I have tried a couple of ways like by using dissect and grok filter but no luck.

Dissect Filter:

 input {
      beats {
        port => 5044
      }
    }

    filter {
    dissect { mapping => { "message" => "%{[@metadata][timestamp]} %{+[@metadata][timestamp]} %{+[@metadata][timestamp]}" } }
        date { match => [ "[@metadata][timestamp]", "MMM dd HH:mm:ss" ] }
     }

Grok filter:

input {
  beats {
    port => 5044
  }
}

filter {
grok {
      match => [ "message", "(?<sourcestamp>(\d){4}-(\d){2}-(\d){2} (\d){2}:(\d){2}:(\d){2},(\d){3})" ]
    }
    date {
      match => [ "sourcestamp" , "yyyy-MM-dd HH:mm:ss,SSS" ]
      target => "@timestamp"
      timezone => "America/Chicago"
}
}

Any help would be appreciated. Thank you!

The timestamp is the first two words of the message, not three. Try

dissect { mapping => { "message" => "%{[@metadata][timestamp]} %{+[@metadata][timestamp]} %{}" } }

You need the trailing %{} to match the rest of the message. And your timestamp format is not "MMM dd HH:mm:ss". Try

    date { match => [ "[@metadata][timestamp]", "yyyy-MM-dd HH:mm:ss" ] }
1 Like

Thank you very much! It's working now.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.