Get the correct timestamp from the log file

Elastic Stack Logstash
1

The log file has its own timestamp and i am trying to replace the logstash timestamp with my timestamp.
I got the _dateparsefailure and _grokparsefailure .

This is my config file :

input
{
file
{

path => "C:\Users\eagasur\logs1.log"
start_position => "beginning"

}

}
filter
{
grok
{
match => [ "message", "%{MONTHDAY:day}/%{MONTHNUM:month}/%{YEAR:year}%:%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} %{ISO8601_TIMEZONE:timezone} %{LOGLEVEL:log_level} %{NUMBER:line:int}" ]
}

ruby
{
code=> "hr=event['hour'].to_i ;
min = event['minute'].to_i ;
sec = event['second'].to_i;
hr_to_sec = hr * 60 * 60;
min_to_sec = min * 60;
total= hr_to_sec + min_to_sec + sec ;
event['time']=total * 1000;
event['difference'] = event['time'].to_i - var1;
var1=event['time'].to_i"
}
mutate
{
add_field => { "timestamp" => "%{day}/%{month}/%{year}:%{hour}:%{minute}:%{second} %{timezone}" }
}
date
{
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
}
}

output
{
stdout
{
codec => rubydebug{}
}
}

2

So those two error tags are already telling you something :slight_smile:

_grokparsefailure is added by any grok filter when it fails to perform a match on a message with the given patterns. So this means your grok filter isn't matching correctly, which means the rest of your filters that rely on those extracted fields are probably not working either.

_dateparsefailure is added by and date filter when it fails to match against the given regex/pattern match. So in your case, your pattern dd/MMM/YYYY:HH:mm:ss Z isn't matching the contents of the timestamp field, most likely because your timestamp field was created with fields extracted by your grok filter, which isn't working either :smile:

Long story short, you need to fix your grok filter. Have you tried using the grok debugger to create a pattern that matches actual lines from your log? I'd highly recommend you start there. Once you've got your grok filter working, problems with the other filters should start disappearing and you can work through anything that doesn't.

Once you've all issues and got a working config, you should consider making use of the @metadata field for all transient fields you don't actually want to add to the documents you store in Elasticsearch. See the blog post about using it here, it has an example you could adapt for your date processing.

3

Yes evenn if i resolved that issue mytimestamp is not targetted to @timestamp. :smiley:

I have done some changes in configuration file.This time i am just focussing on the timestamp .

input
{
file
{
path => "C:\Users\eagasur\logs1.log"
start_position => "beginning"
}
}

filter
{
grok
{
match =>
[
"message", "(?%{MONTHDAY}/%{MONTH}/%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE})"
]
}
date
{
match => ["mytimestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]

}

}
output
{
stdout
{
codec => rubydebug{}
}

elasticsearch
{
cluster => "elastic"
action => "index"
host => "localhost"
index => "act"

}
}

4

Sorry, I'm not sure what you mean by "mytimestamp is not targetted to @timestamp". As it stands, your date filter will write a matched timestamp from the mytimestamp field to the @timestamp field. If you don't want this, you need to set a different target parameter in this filter.

I unfortunately can't help any further. Without seeing your actual log data or errors, I can't tell whether the new config will work or not. Have you tried using the grok debugger tool I linked to in my last reply? This will be the best way of verifying your configuration works short of just running it on the data.

5

This is my log file:

21/Oct/2015:12:28:13 +0530
21/Oct/2015:12:28:13 +0530
21/Oct/2015:12:28:13 +0530
21/Oct/2015:12:28:13 +0530
21/Oct/2015:12:28:13 +0530
21/Oct/2015:12:28:13 +0530

And this is my output:

Logstash startup completed
{
"message" => "21/Oct/2015:12:28:13 +0530\r",
"@version" => "1",
"@timestamp" => "2015-10-21T06:58:13.000Z",
"host" => "7LH4ZR1",
"path" => "C:\Users\eagasur\logs1.log",
"mytimestamp" => "21/Oct/2015:12:28:13 +0530"
}
{
"message" => "21/Oct/2015:12:28:13 +0530\r",
"@version" => "1",
"@timestamp" => "2015-10-21T06:58:13.000Z",
"host" => "7LH4ZR1",
"path" => "C:\Users\eagasur\logs1.log",
"mytimestamp" => "21/Oct/2015:12:28:13 +0530"
}

As logstash generates its own timestamp in the field @timestamp. But i have to use mytimestamp of the old logfile for plotting the graph in Kibana so i am trying to replace the @timestamp with mytimestamp. But its not working.
I have used the same procedure as discussed in the link:

6

@Surbhi_Agarwal from your last message, your date filter is working correctly but you may be missing that the timezone is changed to UTC for technical reason (in short ES time-based indexes are expected by Kibana to use UTC, so logstash enforce using UTC in @timestamp to ensure your event is stored in the correct index and can be found by Kibana)

so 21/Oct/2015:12:28:13 +0530 => 2015-10-21T06:58:13.000Z is the expectect behaviour