Issue with extracting timestamp value from log file

Elastic Stack Logstash
1

Log file 1 format
2019-02-26 03:00:50.000 -05:00 172.11.0.1 LOCAL:DEBUG some message
Log file 2 format
2019-01-12T05:03:04.956-05:00 172.11.0.1 LOCAL:DEBUG some message

I have created a common GROK format for the above files and trying to extract the log timestamp in field called logTimestamp. If I run the files separately one by one it works but if I run them both at the same time using filebeat then it fails. My logstash filter is as below

 grok {
        match => { "message" => ["^%{DATA:date} %{DATA:time} %{DATA:offset} %{GREEDYDATA:msgFragments}$"
    							]
    	
    				}
    	}
    	mutate {
    				add_field => { "doc_type" => "utilization" 
    							   "trans_type" => "xx"
    							   "record_type" => "yy" 
    								"parse_tag" =>  "commonSTRING"
    							 }
    			}
    	if [date] =~ /([12]\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]))(?!T)/ {
    			mutate {
    					
    					add_field => { "logTimestamp" => "%{date}T%{time}%{offset}"}
    					}
    				}
    	else {
    		  mutate {	
    					add_field => { "logTimestamp" => "%{date}"}
    				}	
    		 }
    	date {
        match => [ "logTimestamp", "ISO8601" ]
         }

@magnusbaeck -- can you please help me understand what I may be doing wrong ? thanks in advance !

2

I would suggest a much more specific grok filter.

    grok { match => { "message" => "^(?<date>[0-9\-]{10}[ T][0-9:.]{12}\s*[0-9\-:]{6}) %{IPV4:ip} %{GREEDYDATA:restOfLine}" } }
    date { match => [ "date", ISO8601, "yyyy-MM-dd HH:mm:ss.SSS ZZ" ] }
3

@Badger -- thank you so much ! That regex did the trick! I am new to this scripting, is there any good reference that you can recommend for me to study further ?
Also I had another question ! I had read elsewhere that regex can sometime be costly operation as it tries to compare pattern ! I am new to this GROK scripting and wanted to make sure that I take performance as well in consideration as I iterate over this. Not sure if there is any easy answer to this but any pointers to good source will help! Do you use any tools to monitor how these expressions perform ?
Thanks again for your answer...
regards
rags

4

It can be. This post on the elastic blog has some pointers.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.