Unable to parse date in grok and filters

I have the following message

2021-01-04T00:04:00.8033345+00:00 0HM5E5E28610F:00000001 [INF] MyMessage

What I'm trying to do is to extract each of those fields into a separate column for my index.

This is what I have been using to parse the message in my pipeline

    filter {
    mutate {
      strip => "message"
    }
    grok {
      match => { "message" => "%{DATA:logdate} %{DATA:thread_id} \[%{LOGLEVEL:log.level}\] %{GREEDYDATA:message}" }
    }
    date {
      match => [ "logdate", "yyyy-MM-dd HH:mm:ss.SSSSSSSZZ" ]
      target => "@timestamp"
    }
}

At the end, I'm getting parse errors and my message ends up in Elasticsearch by sending the whole message as a column in my index. The idea is to break the 4 data I have in each line into different columns. Not only GrokParseError I receive, but also issues while converting the date.

Any idea on how to solve it?

I think it should be worked with grok.

%{TIMESTAMP_ISO8601:action} %{NOTSPACE:thread_id} \[%{WORD:log.level}\] %{GREEDYDATA:message}
1 Like

Thank you very much, it could solve the parsing issue now.
Although, I still see 3 issues:

  1. The @logdate field shouldn't exist, as I want to replace the log entry time from the @timestamp, that means, my timestamp should be pointing to January, not the collected date;

  2. The message should contain only the last field in my log entry, expected would be only "msg"

  3. I see that the additional fields I added are giving me an exclamation sign that the field is not cached

Any ideas how to solve the above?

Thanks!

(3). Go to Management, index pattern and refresh field. That is quick way. But you can index template (it look like database schema on mysql/sql).
(2). I think no need replace "message" field. You can remove message and create new field msg
(1). You can use date filter plugin https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match

filter {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:action} %{NOTSPACE:thread_id} \[%{WORD:log.level}\] %{GREEDYDATA:msg}" }
    }

    mutate {
        remove_field => [ "message" ]
    }
}
1 Like

Thanks, that did the trick!

Best Regards.