Elapsed filter but problem in calculate the real diference

Hi all,

I´ve a elapsed filter but the calculation of the elapsed time is based on the timestamp value the occurs when the data enters the Elastic not the value i'm getting from the DB.
I've seen a couple of posts saying to override the timestamp field but i can't make it work.

Here is my configuration:

input {
jdbc {
jdbc_driver_library => "D:\elk\logstash-6.1.1\vendor\ibm\db2jcc.jar"
jdbc_driver_class => "com.ibm.db2.jcc.DB2Driver"
jdbc_connection_string => "XXXXX"
jdbc_user => "XXXX"
jdbc_password => "XXXXX"
tracking_column => date_created
tracking_column_type => "timestamp"
last_run_metadata_path => "D:\elk\logstash-6.1.1\metadata\logstash_jdbc_qld_v2_last_run"
use_column_value => true
schedule => "* * * * *"
statement => "SELECT DATE_CREATED,
DESTINATION_HOST, OPERATION_ID, MESSAGE
FROM COMMUNICATION_LOG
WHERE DATE_CREATED > :sql_last_value "
}
}
filter {
date {
match => [ "date_created", "yyyy-MM-dd HH:mm:ss,SSS"]
target => "@timestamp"
}
if [message_type] == "Req" {
mutate {
add_tag => [ "taskStarted" ]
}
}
if [message_type] == "Res" {
mutate {
add_tag => [ "taskTerminated" ]
}
}
elapsed {
unique_id_field => "operation_id"
start_tag => "taskStarted"
end_tag => "taskTerminated"
timeout => 30
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "eai_qua_log_v2-%{+YYYY.MM.dd}"
document_id =>"%{communication_id}"
}
}

I'm still getting the diference:

  • "@timestamp": "2019-02-01T16:32:00.156Z"
  • "date_created": "2019-02-01T16:30:56.738Z",

And the elapsed time is calculated with the timestamp, so the values are not real.

Can anyone help?

Cheers.

That does not match "yyyy-MM-dd HH:mm:ss,SSS". Change your date filter to

date { match => [ "date_created", "ISO8601" ] }

[quote="bigster, post:1, topic:166760"]
target => "@timestamp"
[/quote]I've changed the date format, but i still get diferences between the date_created and the @timestamp.

"@timestamp": "2019-02-04T13:56:00.233Z"
"date_created": "2019-02-04T13:55:58.515Z"
"elapsed_timestamp_start": "2019-02-04T13:56:00.233Z"

What i miss?

Hi all,

Update for my old comment:
i get _dateparsefailure when i put the "ISO8601"

Anyone knows anything to help?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.