How to read JSON file using logstash and convert String date format to Date time format


(Vignesh Kumar) #1

The Input is like json, Need to get this "TimeStamp": "20170926110036"

{
"Properties": {
"Client Name": "Chubb",
"Portfolio": "Chubb-Transfer"
},
"Capture": [
{
"CaptureGUID": "caa1f5ba-1e93-4926-b3ac-e30d0d9d4cbb",
"HTMLPath": "Captures\C:\",
"ScreenName": "Amdocs CRM - ClearCallCenter - [Console]",
"TimeStamp": "20170926110036"
}
]
}

My Logstash config is as below,
input {
file {
type => "json"
path => "C:/ELK/data/Recordings/*.json"
start_position => beginning
codec => multiline {
pattern => "^{"
negate => "true"
what => "previous"
max_lines => 30000
}
}
}
filter{
date {
match => ["TimeStamp", "yyyyMMddHHmmss"]
target => "TimeStamp"
}

mutate { 
replace => { "message" => "%{message}}" }
gsub => [ 'message','\n','']
}

json { 
source => "message" 
remove_field => ["message"]
}

}

output {
elasticsearch {
index => "test10"
}
stdout { codec => rubydebug }
}

But output of the date format is in string format only it is not changing to date time format.
"Properties" => {"Client Name"=> "Chubb", "Portfolio"=> "Chubb-Transfer" }, "Capture" => [ { "CaptureGUID"=> "caa1f5ba-1e93-4926-b3ac-e30d0d9d4cbb", "HTMLPath"=> "Captures\C:\", "ScreenName"=> docs CRM - ClearCallCenter - [Console]", "TimeStamp"=> 20170926110036"}]


(Magnus Bäck) #2

You have nested fields, so the actual name of your timestamp field is [Capture][0][TimeStamp].


(Vignesh Kumar) #3

Thank you. Please let me know, how to match this in the date filter,
date {
match => ["[Capture][0][TimeStamp]", "yyyyMMddHHmmss"]
target => "TimeStamp"
}
I tried give above the also it is not detecting.


(Magnus Bäck) #4

Please show the resulting event. Use a stdout { codec => rubydebug } output.


(Vignesh Kumar) #5

The resulting is same,
{
"Properties" =>
{"Client Name"=> "Chubb", "Portfolio"=> "Chubb-Transfer" },
"Capture" =>
[
{
"CaptureGUID"=> "caa1f5ba-1e93-4926-b3ac-e30d0d9d4cbb",
"HTMLPath"=> "Captures\C:",
"ScreenName"=> "docs CRM - ClearCallCenter - [Console]",
"TimeStamp"=> "20170926110036"
}
]
}

it is showing the TimeStamp still in string format only.


(Vignesh Kumar) #6

Thanks for the reply it worked
date {
match => ["[Capture][TimeStamp]", "yyyyMMddHHmmss"]
add_tag => "edirtime"
target => "@timestamp"
timezone => "Asia/Kolkata"
locale => "en"
}


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.