Unable to get Date Time from logs into Timestamp

[JonnB]

Hi,

I have gone through previous posts which do not seem to help with my specific issue and its taking a long time to figure out without progress, if you can assist in the group?

I want to get the datetime from my logfile into the timestamp field, an example of the log time is below:

Example Logfile date time format: 20200723T093645+0000

This is my logstash config file which works fine (apart from date issue):

input {
  beats {
    port => 5044
    type => logs
  }
}


filter {
   grok {
      match => { "message" => "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}T%{HOUR:hour}:?%{MINUTE:minute}(?::?%{SECOND:second})?%{ISO8601_TIMEZONE}%{SPACE}%{NOTSPACE:Logtype}%{SPACE} " }
   }

  # Note: if i try to add T between day and hour i get an error

   mutate {
     add_field => { "timestamp" => "%{year}-%{month}-%{day} %{hour}:%{minute}:%{second}" }
   }

    date {  
      match => ["timestamp", "yyyy-MM-dd HH:mm:ss.SSSSSS"]
      target => "@timestamp"
      locale => "en"
      #remove_field => [ "timestamp" ]
    }

}

output {
  if [type] == "logs" {
    elasticsearch { 
      hosts => ["http://192.168.1.56:9200"] 
      index => "logstash-logs-%{+YYYY.MM.dd}"
    }
  }
}

I really do not know what i am doing having gone round in circles so hoping someone get get me past this please

Thanks

[Badger]

@JonnB

match => ["timestamp", "yyyy-MM-dd HH:mm:ss.SSSSSS"]

Remove the .SSSSSS from the pattern. Your timestamp does not have sub-second precision.

That said, I think you are making this way harder than it needs to be. If you changed your grok to use

match => { "message" => "^%{NOTSPACE:timestamp}%{SPACE}%{NOTSPACE:Logtype}%{SPACE} " }

you could match it using

match => ["timestamp", "yyyyMMdd'T'HHmmssZ"]

[JonnB]

Thanks Badger, that was a copy and paste answer!