ES index date format for date/timezone offset

[earlsanchez (ES)]

Hello,
We're using Elasticsearch v7.9.

The date format of our mongod logs changed to this:
2021-09-26T03:23:46.515-0700

Now we're getting "_dateparsefailure" during Logstash processing.

I can't find a specific example of how to handle the timezone offset, can someone please help?

Here is our ES index date format:

"format" : "MM/dd/yyyy HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSSSSS||yyyy-MM-dd'T'HH:mm:ss.SSS'Z'||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd'T'HH:mm:ss.SSSZZZ||yyyy-MM-dd||epoch_millis"

This is from our Logstash conf file to process the date from logs.

    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:date} %{WORD:severity} %{GREEDYDATA:message}" }
    }
    mutate { convert => ["severity","string"] }
    date {
	match => [ "date", "yyyy-MM-dd'T'HH:mm:ss.SSSZZZ" ]
	timezone => "America/Los_Angeles"
	target => "date"
    }
 

This is how Logstash is processing the date field:

"date" => "2021-09-26T03:23:46.515-0700",

TIA

[Badger]

@earlsanchez

"yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"

Change ZZZ to ZZ. The target will then be a LogStash::Timestamp object which will be sent to elasticsearch as a number of milliseconds since the epoch (epoch_millis).