Parse any field that has a specific name

Elastic Stack Logstash
1

Is it possible to parse any field in Json, under any root, that has a unix time format?
I might have in my json file this field
"timestamp": 1554940800000
it might be at any root in the json, with this naming convention of the filed might contain timestamp as name value

right now I'm parsing it using a hardcoded path because I know where that field is in the json, but I need to generalize it, also process all fields if there is more that one.

filter {
  date {
        match => [ "[body][resource][value][timestamp]", "UNIX_MS" ]
        target => "[body][resource][value][timestamp]"
  }
}
2

Do you want the timestamp field overwritten with the parsed date?

3

@Badger Yes, as you can see that I'm parsing that specific field using match and target, but it might be at any root/level and the json might have more than one field with that unix value to be parsed. If there is more than one field they'll all contain the value timestamp in their name at somehow.

4

Will you be flattening the JSON with the script I posted yesterday? That greatly simplifies things.

5

@Badger yes exactly, I'll be flattening the json using the ruby script you posted.

6

Then you can use

    ruby {
        code => '
            event.to_hash.each { |k, v|
                if k =~ /(^|\.)timestamp$/ and v.to_s.to_i == v
                    event.set(k, LogStash::Timestamp.new(Time.at(v/1000)))
                end
            }
        '
    }

which will produce

      "body.input.value.timestamp" => 2019-04-11T00:00:00.000Z,
7

@Badger it didn't work when there is two fields timestamp1 and timestamp2, the ruby code can't detect the field if there is characters after the timestamp value and also when the fields names are 1timestamp 2timestamp

8

You could change the regexp that is matching the field name. What do you mean by 'characters after the timestamp value'?

9

if I have test_timestamp or 1_timestamp etc..
I solved it by adding the .* anchor:

event.to_hash.each { |k, v|
                if k =~ /(^|\.).*timestamp.*$/ and v.to_s.to_i == v
                    event.set(k, LogStash::Timestamp.new(Time.at(v/1000)))
                end
            }
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.