Logstash : convert date to unix time problem

Elastic Stack Logstash
1

logstash verison : 5.0.1
i want to convert the time : 20171122194855.000000480 to unix time
how can i do in logstash filter ? thanks so much.

2

You can use a date filter to parse the string into a timestamp object. Then you should be able to use a piece of Ruby in a ruby filter to convert the timestamp object into an epoch integer.

3

I just had the almost same requirement besides converting into Unix time. The method below converts a very similar data to a timestamp that can be picked up by Logstash:

Data:

20171031235935785

Filter:

date {
  match => [ "SOURCE_FIELD", "YYYYMMddHHmmssSSS" ]
}

Maybe this can give you some ideas.

4

Hi ericohtake,
Thx your reply. I have a try , but not successful.
My source log :
Source=Service Control Manager Category=0 RecordNumber=29488 Message=The WinHTTP. Event=7036 Type=Information TimeGenerated=20171121174525.000000480 Computer=computer_PC LogFile=System

logstash filter file:
filter{
grok {
match => {"message" => "Source=%{GREEDYDATA:Source} Category=%{NUMBER:Category} RecordNumber=%{NUMBER:RecordNumber} Message=%{GREEDYDATA:msg} Event=%{NUMBER:eventId} CategoryString=%{GREEDYDATA:CategoryString} Type=%{GREEDYDATA:Type} TimeGenerated=%{NUMBER:time:int} Computer=%{WORD:shost} LogFile=%{GREEDYDATA:LogFile}"}
remove_field => ["message","host", "@version"]
}
date{
match => ["time", "YYYYMMddHHmmss"]
target => "logdate"
}
ruby{
code =>"
event['logunixtime'] = event['logdate'].to_i
"}
}

The error information:
Failed parsing date from field {:field=>"time", :value=>20171123091554, :exception=>"could not coerce Fixnum to class java.lang.String", :config_parsers=>"YYYYMMddHHmmss", :config_locale=>"default=en_US"}

what's my problem ? is that my grok " TimeGenerated=%{NUMBER:time:int} " not right ???
How can I write this?

5

Hi magnusbaeck,
I have tried, but not successful.
logstash filter file:
filter{
grok {
match => {"message" => "Source=%{GREEDYDATA:Source} Category=%{NUMBER:Category} RecordNumber=%{NUMBER:RecordNumber} Message=%{GREEDYDATA:msg} Event=%{NUMBER:eventId} CategoryString=%{GREEDYDATA:CategoryString} Type=%{GREEDYDATA:Type} TimeGenerated=%{NUMBER:time:int} Computer=%{WORD:shost} LogFile=%{GREEDYDATA:LogFile}"}
remove_field => ["message","host", "@version"]
}
date{
match => ["time", "YYYYMMddHHmmss"]
target => "logdate"
}
ruby{
code =>"
event['logunixtime'] = event['logdate'].to_i
"}
}

So what's my problem ? Thank you so much!

6

what's my problem ? is that my grok " TimeGenerated=%{NUMBER:time:int} " not right ???

Drop :int. In this case you don't want the matched string to be converted to an integer.

Unrelated to your problem but I'm mentioning it anyway: For performance (and in some cases correctness) reasons you don't want to have more than one DATA or GREEDYDATA pattern in a file. In this particular case you can just use a kv filter and skip the grok filter.

7

Thanks your remind！

I have dropped : int but it still "_dateparsefailure"
error information :
: exception => " could not coerce Fixnum to class java.lang.String"

8

Please show an example event produced by Logstash. Use a stdout { codec => rubydebug } output.

9

I adjust the filter file :
filter{
grok {
match => {"message" => "Source=%{GREEDYDATA:Source} Category=%{NUMBER:Category} RecordNumber=%{NUMBER:RecordNumber} Message=%{GREEDYDATA:msg} Event=%{NUMBER:eventId} CategoryString=%{GREEDYDATA:CategoryString} Type=%{GREEDYDATA:Type} TimeGenerated=%{NUMBER:time} Computer=%{WORD:shost} LogFile=%{GREEDYDATA:LogFile}"}
remove_field => ["message","host", "@version"]
}

           mutate {
               
                    convert => {"time", "int"}

}
mutate {
convert => {"time", "string"}
}
date{
match => ["time", "YYYYMMddHHmmss"]
target => "logdate"
}
ruby{
code =>"
event['logunixtime'] = event['logdate'].to_i
"}
}

so it worked. Next I will try to use a kv filter and not use grok filter. thanks for your points

10

Remove the two mutate filters. There's no point in converting the field value back and forth.

1 Like
11

Yes, yes, and it also worked. and filter file changed.
date{
match => ["time", "YYYYMMddHHmmss.SSSSSSSSS"]
target => "logdate"
}

1 Like
12

This is a very valuable tip. Thanks!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.