Unable to get Date Time from logs into Timestamp


I have gone through previous posts which do not seem to help with my specific issue and its taking a long time to figure out without progress, if you can assist in the group?

I want to get the datetime from my logfile into the timestamp field, an example of the log time is below:

Example Logfile date time format: 20200723T093645+0000

This is my logstash config file which works fine (apart from date issue):

input {
  beats {
    port => 5044
    type => logs

filter {
   grok {
      match => { "message" => "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}T%{HOUR:hour}:?%{MINUTE:minute}(?::?%{SECOND:second})?%{ISO8601_TIMEZONE}%{SPACE}%{NOTSPACE:Logtype}%{SPACE} " }

  # Note: if i try to add T between day and hour i get an error

   mutate {
     add_field => { "timestamp" => "%{year}-%{month}-%{day} %{hour}:%{minute}:%{second}" }

    date {  
      match => ["timestamp", "yyyy-MM-dd HH:mm:ss.SSSSSS"]
      target => "@timestamp"
      locale => "en"
      #remove_field => [ "timestamp" ]


output {
  if [type] == "logs" {
    elasticsearch { 
      hosts => [""] 
      index => "logstash-logs-%{+YYYY.MM.dd}"

I really do not know what i am doing having gone round in circles so hoping someone get get me past this please


Remove the .SSSSSS from the pattern. Your timestamp does not have sub-second precision.

That said, I think you are making this way harder than it needs to be. If you changed your grok to use

match => { "message" => "^%{NOTSPACE:timestamp}%{SPACE}%{NOTSPACE:Logtype}%{SPACE} " }

you could match it using

match => ["timestamp", "yyyyMMdd'T'HHmmssZ"]

Thanks Badger, that was a copy and paste answer! :smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.