Timestamp => @timestamp

Elastic Stack Logstash
1

Hi ,
I have specific log in a file log:
Tue May 2 03:54:22 2017

I use a grok filter to extract the timestamp into a field of its own called "datestamp"
"datestamp" => "Tue May 2 03:57:00 2017 ", "@timestamp" => 2017-05-15T14:25:05.490Z,
My filter here ,
{filter { date { match => ["datestamp" , "yyyy-MM-dd'T'HH:mm:ss.SSSZ"] target => "@timestamp" } }
i know how to replace the @timestamp by datestamp, but it doesn't work (dateparsefailure). I think it's my match , because the date format is not that , right ?

2

Correct. Your date pattern doesn't even resemble what your timestamp actually looks like. Note that the date filter logs what it's having trouble with.

4

Okay thanks,
What date format can match here ?
Where can i find the date filter logs ?

5

What date format can match here ?

The date filter documentation lists which tokens are available. "EEE MMM d HH:mm:ss YYYY" and "EEE MMM dd HH:mm:ss YYYY" should work in your case.

Where can i find the date filter logs ?

In the Logstash log file, typically found in /var/log/logstash.

6

It doesn't work :frowning: , dateparsefailure again
Where can i find date filter logs for test file, because I test this file with logstash-f, so it doesn't write
in/ var/log/logstash...

7

Where can i find date filter logs for test file, because I test this file with logstash-f, so it doesn't write
in/ var/log/logstash...

In that case Logstash should dump the logs to stdout.

8

It just said "_dateparsefailure

9

I'm pretty sure it said a lot more than that. To save everyone's time, please copy/paste the command you're running and the full output you get.

10

So , my output test of run.conf file;

output { stdout { codec => rubydebug {} } }

For test it, I run this

/usr/share/logstash/bin/logstash -f run.conf

My date filter currently:

 date {
   match => ["datestamp" , "EEE MMM d HH:mm:ss YYYY"]
   target => "@timestamp"
 }

And my output ;

{
"path" => "/var/log/saprouterlog/saprouterlog.txt_a_20170502_035422- 20170502_072722",
"datestamp" => "Tue May 2 03:59:22 2017 ",
"@timestamp" => 2017-05-17T08:50:03.584Z,
"@version" => "1",
"host" => "elkf",
"sap_session" => "C106/-",
"message" => "Tue May 2 03:59:22 2017 CONNECT FROM C106/- 172.16.58.19 9/22502",
"type" => "saplog",
"sap_action" => "CONNECT FROM ",
"sap_ipsource" => "172.16.58.199",
"tags" => [
[0] "saprouter1",
[1] "_dateparsefailure",
[2] "connect from"
],
"sap_port" => "/22502"
}
^C10:50:04.721 [SIGINT handler] WARN logstash.runner - SIGINT received. Shuttin g down the agent.
10:50:04.795 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>" main"}
^C10:50:04.889 [SIGINT handler] FATAL logstash.runner - SIGINT received. Termina ting immediately..

11

It looks like you have a trailing space in the datestamp field, which is not accounted for in the pattern. Correct the pattern used to extract this field and I do not see why it would not work.

12

Thanks for your reply, but I try that with 1 more space between MMM and d and it didnt works.

date {
match => ["datestamp" , "EEE MMM d HH:mm:ss YYYY"]
target => "@timestamp"
}

13

I am referring to the space at the end off the string.

14

Okay , it didnt work but it dont said me _dateparsefailure

{
"path" => "/var/log/saprouterlog/saprouterlog.txt_a_20170510_060630-20170510_102448",
"datestamp" => "Wed May 10 06:12:51 2017 ",
"@timestamp" => 2017-05-10T04:12:51.000Z,
"@version" => "1",
"host" => "elkf",
"sap_session" => "C300/-",
"message" => "Wed May 10 06:12:51 2017 CONNECT FROM C300/- 172.16.58.199/55352",
"type" => "saplog",
"sap_action" => "CONNECT FROM ",
"sap_ipsource" => "172.16.58.199",
"tags" => [
[0] "saprouter1",
[1] "connect from"
],
"sap_port" => "/55352"

My filter date after add the space:

 date {
   match => ["datestamp","EEE MMM d HH:mm:ss YYYY "]
   target => "@timestamp"
 }

}
}

15

That seems to work. Remember that @timestamp is always in UTC. Instead of adding the space to the date filter pattern, I would recommend modifying the extraction pattern to get rid of it instead.

16

Ah yes , i forgot it :slight_smile:
We can change UTC right ?

17

Timestamps in Elasticsearch are assumed to be in UTC, and a lot of functionality, including Kibana, relies on this. Trying to change this is therefore to ask for trouble.

18

Ok, Thank you very much for your help Christian and thanks magnus for your great help :slight_smile: