Match timestamp and assign it as @datetime field


(Emmanuel Chanson) #1

I have the following array:
{
"_index": "index_json",
"_type": "json",
"_id": "AVgzYPhrIoaQPqcpVgK8",
"_score": null,
"_source": {
"path": "/tmp/CSV/data.json",
"@timestamp": "2016-11-05T07:25:41.023Z",
"jobs": {
"name": "Onboarding_Deploy_E1DEV4",
"builds": {
"duration": 867357,
"number": 2225,
"timestamp": 1476977583380
}
},
"@version": "1",
"host": "ks3309573.kimsufi.com",
"type": "json"
},
"fields": {
"@timestamp": [
1478330741023
]
},
"sort": [
1478330741023
]
}

I need to match the field "timestamp": 1476977583380 in builds and assign it to @datetime.

I tried the following config in filter but logstat can not start:

filter {
split { field => "[jobs]" }
split { field => "[jobs][builds]" }
date {
** match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z” ]**
** remove_field => "timestamp"**
** }**
}

BR


(Emmanuel Chanson) #2

Tried this but no success:
date {
match => ["timestamp", "UNIX_MS"]
target => "@timestamp"
remove_field => "timestamp"
}

An example:
{
"path" => "/tmp/CSV/data.json",
"@timestamp" => 2016-11-05T09:06:27.205Z,
"jobs" => {
"name" => "Onboarding_Sanity_E1QA",
"builds" => {
"duration" => 1933,
"number" => 176,
"timestamp" => 1471631216231
}
},
"@version" => "1",
"host" => "ks3309573.kimsufi.com",
"type" => "json"
}


(Magnus Bäck) #3

Why are you using the pattern "dd/MMM/YYYY:HH:mm:ss Z" when your timestamp doesn't even resemble that format? I'm guessing the your timestamp are milliseconds since the epoch (i.e. 1476977583380 is Oct 20 at 15:33 UTC). Use the "UNIX_MS" pattern instead.

Secondly, it appears the timestamp field is nested two levels down and this needs to be taken into account. This should work:

date {
  match => ["[jobs][builds][timestamp]", "UNIX_MS"]
  remove_field => ["[jobs][builds][timestamp]"]
}

(Emmanuel Chanson) #4

Thanks a lot, it is the solution.

Regards

Emmanuel


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.