GROK Date/time filter not working

Elastic Stack Logstash
1

Getting logstash failure when parsing the date/time stamp from my log. This seems to work in the GROK Debugger but fails when logstash filters it. Can someone please help to filter the date/time stamp into "date"?

In GROK Debugger:

2020-11-18 13:55:27,797 WARN 34.99.115.241 WINDOWS_10-CHROME8_86.0.4240.198 87b3bf74-0d37-4c3b-b6fc-9aaade95d94e 70055-rc - net.antidot.fluidtopics.server.shared.DevResourcesHelper - Application is not running in dev mode 

%{TIMESTAMP_ISO8601:date} %{LOGLEVEL:loglevel} %{IP:client_ip} %{GREEDYDATA:user_agent} %{DATA:session_id} %{DATA:tenant} - %{GREEDYDATA:logger} - %{GREEDYDATA:exception}

{
  "date": [
    "2020-11-18 13:55:27,797"
  ],
  "loglevel": [
    "WARN"
  ],
  "client_ip": [
    "34.99.115.241"
  ],
  "user_agent": [
    "WINDOWS_10-CHROME8_86.0.4240.198"
  ],
  "session_id": [
    "87b3bf74-0d37-4c3b-b6fc-9aaade95d94e"
  ],
  "tenant": [
    "70055-rc"
  ],
  "logger": [
    "net.antidot.fluidtopics.server.shared.DevResourcesHelper"
  ],
  "exception": [
    "Application is not running in dev mode "
  ]
}

Here is the error from my logstash logs:

[2020-11-20T14:59:31,797][WARN ][logstash.outputs.amazonelasticsearch][main][896acf02a7b1749acbb7aafd96647933db66fb31a5f92811067761301c329795] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ct-applogs-2020.11.20", :_type=>"_doc", :_routing=>nil}, #<LogStash::Event:0x583d1dcd>], :response=>{"index"=>{"_index"=>"ct-applogs-2020.11.20", "_type"=>"_doc", "_id"=>"d6Ti53UBivrH6_HXV1co", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [date] of type [date] in document with id 'd6Ti53UBivrH6_HXV1co'. Preview of field's value: '2020-11-20 14:59:29,371'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2020-11-20 14:59:29,371] with format [yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

Here is the mapping in Elasticsearch:

        "date" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          },
          "format" : "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
2

Your grok is working, but the mapping cannot parse the milliseconds part of the date field. You would have to fix your mapping.

3

@Badger Thank you, updating the date format in the index mapping worked:

"format" : "yyyy-MM-dd HH:mm:ss,SSS||yyyy-MM-dd||epoch_millis"
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.