Logstash date filter, kibana index template warning in logs

Elastic Stack Logstash
1

Hi,

Original log, these dates are in the format 20210504-15:40:02.601 and I have a date filter to convert that in my logstash.yml. I am doing the same filter for both the SendingTime and the TransactTime.

if [layers][fix][fix_fix_SendingTime] {
      date {
      # 20210426-17:47:27.052
      match => [ "[layers][fix][fix_fix_SendingTime]", "yyyyMMdd-HH:mm:ss.SSS" ]
      target => "[layers][fix][fix_fix_SendingTime]"
      #target => "sendingtime"
      }
    }

I am trying to use an index template (7.10) to map a couple fields to date format. When I do that I get the error below but the field (SendingTime) looks like it did map correctly. As an example, I attached a picture of the TransactTime field which is still being classified by kibana as text.

Screen Shot 2021-05-04 at 12.01.13 PM
Screen Shot 2021-05-04 at 12.01.13 PM1056×58 7.21 KB
Screen Shot 2021-05-04 at 12.01.24 PM
Screen Shot 2021-05-04 at 12.01.24 PM1054×66 7.62 KB

[WARN ] 2021-05-04 16:55:53.090 [[wei_pipeline_1]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"fix-test-2021.05.04", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x6faab899>], :response=>{"index"=>{"_index"=>"fix-test-2021.05.04", "_type"=>"_doc", "_id"=>"dldOOHkBZU4iWKXXtezu", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [layers.fix.fix_fix_SendingTime] of type [date] in document with id 'dldOOHkBZU4iWKXXtezu'. Preview of field's value: '20210504-16:55:50.270'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [20210504-16:55:50.270] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}

2

That looks like it did not go through the date filter, since the date filter should have parsed and overwritten it.

3

Hi,

I used the add_tag in the date plugin and can see the tags are applied.

Thanks!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.