Handling epoch timestamps in logstash filter


(Deepti) #1

Hello,

I need help to convert epoch timestamp to regular date time stamp. But this epoch timestamp (CrashTime) exists in the header of the log file under metadata section. I am storing all these metadata values in @@Metadata parameter and applying It to each log line. How can I use CrashTime value as timestamp by converting it to regular date time? Please check the code below:

[Metadata]

DType: XYZ
SNumber: SSKD293
SystemVersion:1290N
BuildType: debug
CrashTime: 1555363226
state: e
LType: et
Reason: 154659DED7
Utc: P280KSL01

[Events]
01-07 11:01:51.501 3602 3616 I start: LogUpload
01-07 11:01:51.509 3602 10960 I end : ,empty


#2
    mutate { add_field => { "someField" => "1552363226" } }
    date { match => [ "someField", "UNIX" ] }

will result in

"@timestamp" => 2019-03-12T04:00:26.000Z,

Just change the field name to match your event.


(Deepti) #3

Thanks for the reply Badger.
So my code for adding metadata looks like this:

if [message] == "[Metadata]" or [message] == "[Events]" or [message] =~ /^$/
        {
             drop{}
        } else {
        if [message] =~ /^[A-Za-z]+:/ {
            grok { match =~ ["message", "^(?<key>[^:]+):\s*%{GREEDYDATA:value}"] }
            ruby {
                init => '
                    @@collectingMetadata = false
                '
                code => '
                    unless @@collectingMetadata
                        @@metadata = {}
                        @@collectingMetadata = true
                    end
                    @@metadata[event.get("key")] = event.get("value")

                '
            }
            drop {}
          } else {
            ruby {
                code =>'
                    @@collectingMetadata = false
                    event.set("metadata", @@metadata)
                '
            }
        }
        }

So I get CrashTime as one of the metadata keys. Is there a way I can use this conversion in this code? How to use it in ruby? Is it possible?

Need to add it here as I want to apply that epoch time to all the log lines from this file. Please let me know what you think.


#4

You should be able to use

date { match => [ "[metadata][CrashTime]", "UNIX" ] }