Date filter not working [Solved]

Elastic Stack Logstash
1

Hi all,

my first post on logstash forum :spy:

i'm reading events from database using logstash. The events are indexed in elasticsearch.
There is a event_timestamp in the table , the column name is "EVENT_TIMESTAMP" (ORACLE database) and the column type is varchar2(100).

filter {
date {
match => [ "EVENT_TIMESTAMP" , "yyyy-MM-dd HH:mm:ss.SSS" ]
target => "@timestamp"
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-broker"
workers => 8
}

stdout {
    codec => rubydebug
}

}

running logstasth the output is the following:

:\Laboratorio\LogStash\logstash-2.3.1\bin>logstash -f ..\AcoreanaLab_Broker.con

o/console not supported; tty will not be manipulated
ettings: Default pipeline workers: 4
ipeline main started

          "wmb_msgkey" => "414d51204d5142524b45534230314120074b945320002b03

414d51204d5142524b455342303141206524105725ea753f",
"has_bitstream" => "N",
"has_exception" => "N",
"has_userdata" => "N",
"event_type" => "transactionStart",
"event_name" => "transactionStart",
"event_srcaddr" => nil,
"broker_name" => "BRKESB01A",
"broker_uuid" => "f0a1f19e-b5c7-4690-aa09-7380652049ab",
"exgrp_name" => "eg_batch_01",
"exgrp_uuid" => "11a47355-4b01-0000-0080-959c9f749b12",
"msgflow_name" => "BPM_TimedSendCheckAbsence",
"msgflow_uuid" => "399a3990-4c01-0000-0080-ee0f1a3b7f9d",
"appl_name" => nil,
"appl_uuid" => nil,
"library_name" => nil,
"library_uuid" => nil,
"node_name" => "Timeout Notification",
"node_type" => "ComIbmTimeoutNotificationNode",
"detail" => nil,
"terminal_name" => nil,
"key_fld_1_nm" => nil,
"key_fld_2_nm" => nil,
"key_fld_3_nm" => nil,
"key_fld_4_nm" => nil,
"key_fld_5_nm" => nil,
"event_timestamp" => "2016-05-18 00:02:34.370",
"local_transaction_id" => "37b2b40a-2615-497c-b41c-11e716ccccf2-3",
"parent_transaction_id" => nil,
"global_transaction_id" => nil,
"data_type" => nil,
"payload_size" => 0,
"@version" => "1",
"@timestamp" => "2016-05-18T14:38:41.356Z",
"type" => "MonitoringBroker",
"tags" => [
[0] "MonitoringBroker"
]

ipeline main has been shutdown
topping pipeline {:id=>"main"}
he signal HUP is in use by the JVM and will not work correctly on this platform

:\Laboratorio\LogStash\logstash-2.3.1\bin>

event_timestamp = 2016-05-18 00:02:34.370"

but

@timestamp = "2016-05-18T14:38:41.356Z"

why ? what config changes should i make ?

Best regards

Rui Madaleno

1 Like
2

Found the problem :slight_smile:

changed the value of the field in "match" to lowercase

not-working

filter {
date {
match => [ "EVENT_TIMESTAMP" , "yyyy-MM-dd HH:mm:ss.SSS" ]
target => "@timestamp"
}
}
working

filter {
date {
match => [ "event_timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
target => "@timestamp"
}
}

2 Likes