Unable to convert timestamp to Date field

Elastic Stack Logstash
1

I am trying to convert a string field called timestamp with value 1510722000000 in date format field in Logstash. My sole purpose is to visualize data in kibana using this date field. I tried using the date filter but it does not create the target field for me. Can anyone tell me How can I achieve this

My date filter looks like this
date {
timezone => "UTC"
match => ["timestamp", "UNIX_MS"]
target => "@timestamp1"
}

2

This looks okay. Please show

  • your full configuration,
  • an example input message,
  • the output from a stdout { codec => rubydebug } output plugin.
3 
      "tags" => [
    [0] "multiline",
    [1] "_dateparsefailure"
],
      "path" => "C:\\Users\\186\\Downloads\\xml_dev\\27.xml",
"@timestamp" => 2018-02-16T16:08:33.969Z,
4

my Configuration file

input {
file {
path => "C:\Users\186181152\Downloads\xml_dev\27.xml"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "measurement.$"
what => "previous"
}

}
}

filter
{
xml
{
source => "message"
target => "doc"
store_xml => false
xpath => ["//measure/@measure","bt_name"]
xpath => ["//measure/@aggregation","aggregation_type"]
xpath => ["//measurement/@count","count"]
xpath => ["//measurement/@timestamp","timestamp"]
xpath => ["//measurement/@min","min"]
xpath => ["//measurement/@sum","sum"]
}
date {
timezone => "UTC"
match => ["timestamp", "UNIX_MS"]
target => "@timestamp1"
}
if "Count" in [aggregation_type]
{
mutate {add_field => {"aggr_count" => "%{aggregation_type}"}}
}
else if "95th Percentile" in [aggregation_type] {
mutate {add_field => {"aggr_95" => "%{aggregation_type}"}}
}

mutate 
{
convert => {"count" => "integer"
			"min" => "integer"
			"max" => "integer"
			"sum" => "integer"
}
remove_field => ["aggregation_type"]	
}

}

output
{
elasticsearch
{
action => "index"
index => "bt2"
hosts => "localhost:9200"
workers => 1
}
stdout
{
codec => rubydebug
}
}

5

My Sample input
< measurement timestamp="1510722000000" blah blah>

6

Hi @magnusbaeck

The input value in timestamp is 1510722000000 . When I convert this to human readable format using online tools I get 11/16/49842 @ 8:00am (UTC) . Can this be the reason for parse failure ?

7

Please show the whole message, not just the tags, path, and @timestamp fields.

8

The input value in timestamp is 1510722000000 . When I convert this to human readable format using online tools I get 11/16/49842 @ 8:00am (UTC) . Can this be the reason for parse failure ?

Perhaps those online tools assume it seconds from the epoch when your timestamp is actually milliseconds from the epoch.

9

@magnusbaeck Hi, Hope this helps

      "tags" => [
    [0] "multiline",
    [1] "_dateparsefailure"
],
      "path" => "C:\\Users\\186\\Downloads\\xml_dev\\27.xml",
"@timestamp" => 2018-02-20T18:04:05.641Z,
       "avg" => [
    [ 0] "5262.85302734375",
    [ 1] "4866.47314453125",
    [ 2] "5032.392822265625",
    [ 3] "4062.9139404296875",
    [ 4] "4947.111328125",
    [ 5] "4758.715087890625",
    [ 6] "3081.375244140625",
    [ 7] "4437.6788330078125",
    [ 8] "7593.035888671875",
    [ 9] "15168.4794921875",
    [10] "4901.086669921875",
    [11] "4973.13232421875",
    [12] "5004.08642578125",
    [13] "4736.31689453125",
    [14] "4944.011474609375",
    [15] "4908.229248046875",
    [16] "4778.435302734375",
    [17] "4489.117919921875"
],
       "min" => [
    [ 0] 5262,
    [ 1] 4866,
    [ 2] 5032,
    [ 3] 4062,
    [ 4] 4947,
    [ 5] 4758,
    [ 6] 3081,
    [ 7] 4437,
    [ 8] 7593,
    [ 9] 15168,
    [10] 4901,
    [11] 4973,
    [12] 5004,
    [13] 4736,
    [14] 4944,
    [15] 4908,
    [16] 4778,
    [17] 4489
],
  "@version" => "1",
      "host" => "B433",
   "bt_name" => [
    [0] "test"
],
 "timestamp" => [
    [ 0] 1518044400000,
    [ 1] 1518087600000,
    [ 2] 1518109200000,
    [ 3] 1518130800000,
    [ 4] 1518174000000,
    [ 5] 1518195600000,
    [ 6] 1518217200000,
    [ 7] 1518260400000,
    [ 8] 1518368400000,
    [ 9] 1518411600000,
    [10] 1518433200000,
    [11] 1518454800000,
    [12] 1518476400000,
    [13] 1518519600000,
    [14] 1518541200000,
    [15] 1518562800000,
    [16] 1518606000000,
    [17] 1518627600000
]

}
{
"@version" => "1",
"host" => "B433",
"path" => "C:\Users\186\Downloads\xml_dev\27.xml",
"@timestamp" => 2018-02-20T18:04:05.642Z,
"message" => " "
}
{
"@version" => "1",
"host" => "B433",
"path" => "C:\Users\186\Downloads\xml_dev\27.xml",
"@timestamp" => 2018-02-20T18:04:05.643Z,
"message" => " "
}
{
"@version" => "1",
"host" => "B433",
"path" => "C:\Users\186\Downloads\xml_dev\27.xml",
"@timestamp" => 2018-02-20T18:04:05.643Z,
"message" => " "
}
{
"@version" => "1",
"host" => "B433",
"path" => "C:\Users\186\Downloads\xml_dev\27.xml",
"@timestamp" => 2018-02-20T18:04:05.644Z,
"message" => " "
}

10

The timestamp field contains an array of timestamps. Which one do you want to parse?

11

I want to parse all timestamps so I can use them to visualize in Kibana

12

Then you can't use the date filter since it doesn't process arrays of input strings. It should be easy to do with a ruby filter, however, but I don't have time to give an example.

13

Can you direct me to similar post where I can try to explore what I am trying to achieve?

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.