Logstash date field error

Elastic Stack Elasticsearch
1

I have log files from where I've to query documents through the date field. Here is the log date format.
2020-06-10 14:01:26
I've defined a grok pattern like below in Logstash for the log.

filter {
    grok {
           match => {"message" => "%{YEAR:YEAR}-%{MONTHNUM:MONTHNUM}-%{MONTHDAY:MONTHDAY} %{TIME:time} ........."
          }}
    mutate {
            add_field => {
              "log_timestamp" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}"
 }}

Date field log_timestamp is working as expected but I've been receiving Logstash error message frequently because of this grok pattern. In a day, if there's a log document of around 100 then 10 logs will be discarded because of this error. Is there any better way to parse the log file and get the date field?

Error msg:

[2020-06-10T06:43:18,756][WARN ][logstash.outputs.elasticsearch][log] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"log-2020-06", :_type=>"_doc", :routing=>nil}, #LogStash::Event:0x7f4158ff], :response=>{"index"=>{"_index"=>"log-2020-06", "_type"=>"_doc", "_id"=>"KSSvm3IBbBjRTTrvQZx8", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [log_timestamp] of type [date] in document with id 'KSSvm3IBbBjRTTrvQZx8'. Preview of field's value: '%{YEAR}-%{MONTHNUM}-%{MONTHDAY}'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [%{YEAR}-%{MONTHNUM}-%{MONTHDAY}] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

2

Can anyone suggest on this?

3

I don't see anything wrong with your parsing. Nonetheless, if you are getting errors, I would try using explicit regular expressions, to make sure that you are getting what you expect.

For example, if your pattern is always exactly as you quoted, you could use this to parse it:

filter {
  grok {
    match => { "message" => "(?<YEAR>\d{4})-(?<MONTHNUM>\d{2})-(?<MONTHDAY>\d{2})\s(?<HOUR>\d{2}):(?<MINUTE>\d{2}):(?<SECOND>\d{2})" }
  }
}

These matches are stricter than the grok patterns you quoted, but maybe this can help to narrow down the problem.

4

Thank you for your reply.

I've followed your advice but still having the same issue. Here is the error message,

"reason"=>"failed to parse date field [%{YEAR}-%{MONTHNUM}-%{MONTHDAY}] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

5

What's the content of the message field for the message that gives you the error?

6

Error is related to log_timestamp field, date field.
I found a similar problem reported on Github too.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.