_dateparsefailure, can't debug further

[marinipete]

So I've been creating logstash pipelines for a while now and this is the first time I came across an issue I couldn't find a solution online or debug further, so I decided to post here.

I'm getting '_dateparsefailure' in my output tags while trying to parse a simple ISO8601 timestamp from a json I'm receiving from AWS Kinesis, that's my pipeline:

input {
    kinesis {
            kinesis_stream_name => "es-tractor-osb-data-prd"
            application_name => "logstash-kinesis-poc-v2"
            region => "us-east-1"
            profile => "default"
            codec => json { }
        }
}
filter {
    date {
        match => ["@timestamp", "ISO8601"]
        locale => "en-US"
        timezone => "America/Sao_Paulo"
        target => "newtimestamp"
    }
}

output {
  if "erro" in [metricType] {
      if [component] == "CheckoutV2/CheckoutSOAPV2" {
         file {
             path => "/home/ec2-user/logstash/newosb-output-kinesis-all-to-emr-error"
             enable_metric => false
          }
  }
  }
  if [component] == "CheckoutV2/CheckoutSOAPV2" {
 file {
     path => "/home/ec2-user/logstash/newosb-output-kinesis-all-to-emr"
     enable_metric => false
  }
  }
}

Sample input:

{"component":"CheckoutV2/CheckoutSOAPV2","@timestamp":"2019-09-04T22:16:00.230Z"}

What I find strange is that this works:

[root]$echo "2019-09-04T22:16:00.230Z" | bin/logstash -e 'input { stdin {} } filter { date { match => [ "message", "ISO8601"] locale => "en-US" timezone => "America/Sao_Paulo" target => "newtimestamp" } }'
{
        "@version" => "1",
            "host" => "ip-10-2-9-142.ec2.internal",
    "newtimestamp" => 2019-09-04T22:16:00.230Z,
         "message" => "2019-09-04T22:16:00.230Z",
      "@timestamp" => 2019-09-04T23:10:17.979Z
}

I've already set logstash's log do trace but there is not a single dateparsefailure hint in the logs.
Trying to match "YYYY-MM-dd'T'H:mm:ss.SSS'Z'" results in the same problem. No 'newtimestamp' field comes up in my output file and I get the dateparsefailre in the tags.

Any help would be greatly appreciated.

[Badger]

A date filter parses a string. It cannot parse a LogStash::Timestamp. The @timestamp field is special. If a json filter sees a field called @timestamp it will try to parse it into a LogStash::Timestamp...

input { generator { count => 1 lines => [ '{"component":"CheckoutV2/CheckoutSOAPV2","@timestamp":"2019-09-04T22:16:00.230Z"}' ] } }
filter { json { source => "message" } }
output { stdout { codec => rubydebug } }

results in

"@timestamp" => 2019-09-04T22:16:00.230Z,

Note that there are no double quotes around the value of @timestamp, which is how you can tell it is a LogStash::Timestamp and not a string.

[marinipete]

Hello Badger, thanks for the insight. I did consider this before, so I tried to copying (with mutate filter) the value of @timestamp to another field, then applying the date parse on that field, but got the same result.

Also, the json I'm receiving from Kinesis has another field called timestamp (without the '@'), the value of this field has no double quotes, so I guess this is the field Logstash considering as a timestamp ?

Input json:
{"flowId":"data","host":"data","metadata":"data","componentType":"data","metricType":null,"messageType":"data","component":"data","@timestamp":"2019-09-04T22:57:19.233Z","payload":null,"way":"data","instanceId":"data","timestamp":1567637837587,"@version":"1"}

[Badger]

@marinipete

I tried to copying (with mutate filter) the value of @timestamp to another field, then applying the date parse on that field, but got the same result.

It's type will not change when you copy it. You could mutate+convert it to be a string, and then parse it, but why would you want to that? It is already parsed!

[marinipete]

I tried using the mutate+convert strategy but still got _dateparsefailure:

filter {
    mutate {
        copy => {"@timestamp" => "timestamp" }
        convert => { "timestamp" => "string" }    
    }

    date {
        match => ["timestamp", "ISO8601"]
        locale => "en-US"
        timezone => "America/Sao_Paulo"
        target => "timestamp"
    }
}

but why would you want to that? It is already parsed!

Basically, I'm sending this output to a Big Data Platform that is expecting to receiving this input:

{"@timestamp":"2019-08-26T23:48:31.931Z","timestamp":"2019-08-26T20:48:30.347-03:00"}

So I'm trying to convert the bellow, into the above:

{"@timestamp":"2019-08-26T23:48:31.931Z","timestamp":1567637837587}

[Badger]

A mutate filter does operations in a fixed order and convert comes before copy, so the timestamp field would not exist when the convert executes, so it is a no-op. Split the mutate into two filters.

[marinipete]

Wow, didn't know about the fixed order of the operations.
Thanks a lot for the help Badger, that did it!