Hi,
In my current architecture we are reading csv files through Filebeat and sending it to Logstash and then finally ingesting data into Elastic. But date type is coming as string even tough we mapped as date in logstash.
Below is my logstash config
input { beats { port => "5044" } }
filter {
if [type] == "sdc_dev_searchsdcproject" { csv { separator => "," columns => [ "Version" ,"Filename" ,"Method File Created" ,"Author" ,"Update Sample Weight" ,"Total Gas Flow" ,"Sampling Rate" ,"Save Data Rate" ,"DMDT Window (min.)" ,"Min. DMDT Stability Duration (min.)" ,"Max. DMDT Stability Time (min.)" ] } } mutate {convert => ["Version", "float"]} mutate {convert => ["Filename", "string"]} mutate {convert => ["Author", "string"]} mutate {convert => ["Update Sample Weight", "string"]} mutate {convert => ["Total Gas Flow", "integer"]} mutate {convert => ["Sampling Rate", "integer"]} mutate {convert => ["Save Data Rate", "integer"]} mutate {convert => ["DMDT Window (min.)", "integer"]} mutate {convert => ["Min. DMDT Stability Duration (min.)", "integer"]} mutate {convert => ["Max. DMDT Stability Time (min.)", "integer"]}
date{
match => ["Method File Created","yyyy-MM-dd HH:mm:ss Z","ISO8601"]
target => "Method File Created"
}
}
output {
if [type] == "sdc_dev_searchsdcproject" { elasticsearch { hosts => [ "amrvpce1a02086.pfizer.com" ] index => "sdc_dev_search" document_type => "doc_type_sdc_dev_search" user => "elastic" password =>"changeme"}}
}
and below is my input data
|Version|Filename|Method File Created|Author|Update Sample Weight|Total Gas Flow|Sampling Rate|Save Data Rate|DMDT Window (min.)|Min. DMDT Stability Duration (min.)|Max. DMDT Stability Time (min.)
|1.01|C:\DVS-Advantage\method\0_90 full in 10s 200cc 25C dm 08 no tare.sao|5/9/2000|FCNSAN-G323|OFF|200|1|300|5|120|1440|
|1.02|C:\DVS-Advantage\method\0_90 full.sao|5/10/2000|FCNSAN-G241|OFF|450|4|210|8|243|1234|
|1.03|C:\DVS\method\abc.sao|5/11/2000|FCNSAN-G159|ON|700|7|120|11|366|1028|
Please let me know how I can ingest date as date and not as string.
Regards,
Vijay