Timestamp => @timestamp

Elastic Stack Logstash
1

Hi ,
I have specific log in a file log:
Tue May 2 03:54:22 2017

I use a grok filter to extract the timestamp into a field of its own called "datestamp"
"datestamp" => "Tue May 2 03:57:00 2017 ", "@timestamp" => 2017-05-15T14:25:05.490Z,
My filter here ,
{filter { date { match => ["datestamp" , "yyyy-MM-dd'T'HH:mm:ss.SSSZ"] target => "@timestamp" } }
i know how to replace the @timestamp by datestamp, but it doesn't work (dateparsefailure). I think it's my match , because the date format is not that , right ?

2

Correct. Your date pattern doesn't even resemble what your timestamp actually looks like. Note that the date filter logs what it's having trouble with.

1 Like
4

Okay thanks,
What date format can match here ?
Where can i find the date filter logs ?

5

What date format can match here ?

The date filter documentation lists which tokens are available. "EEE MMM d HH:mm:ss YYYY" and "EEE MMM dd HH:mm:ss YYYY" should work in your case.

Where can i find the date filter logs ?

In the Logstash log file, typically found in /var/log/logstash.

1 Like
6

It doesn't work :frowning: , dateparsefailure again
Where can i find date filter logs for test file, because I test this file with logstash-f, so it doesn't write
in/ var/log/logstash...

7

Where can i find date filter logs for test file, because I test this file with logstash-f, so it doesn't write
in/ var/log/logstash...

In that case Logstash should dump the logs to stdout.

1 Like
8

It just said "_dateparsefailure

9

I'm pretty sure it said a lot more than that. To save everyone's time, please copy/paste the command you're running and the full output you get.

1 Like
10

So , my output test of run.conf file;

output { stdout { codec => rubydebug {} } }

For test it, I run this

/usr/share/logstash/bin/logstash -f run.conf

My date filter currently:

 date {
   match => ["datestamp" , "EEE MMM d HH:mm:ss YYYY"]
   target => "@timestamp"
 }

And my output ;

{
"path" => "/var/log/saprouterlog/saprouterlog.txt_a_20170502_035422- 20170502_072722",
"datestamp" => "Tue May 2 03:59:22 2017 ",
"@timestamp" => 2017-05-17T08:50:03.584Z,
"@version" => "1",
"host" => "elkf",
"sap_session" => "C106/-",
"message" => "Tue May 2 03:59:22 2017 CONNECT FROM C106/- 172.16.58.19 9/22502",
"type" => "saplog",
"sap_action" => "CONNECT FROM ",
"sap_ipsource" => "172.16.58.199",
"tags" => [
[0] "saprouter1",
[1] "_dateparsefailure",
[2] "connect from"
],
"sap_port" => "/22502"
}
^C10:50:04.721 [SIGINT handler] WARN logstash.runner - SIGINT received. Shuttin g down the agent.
10:50:04.795 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>" main"}
^C10:50:04.889 [SIGINT handler] FATAL logstash.runner - SIGINT received. Termina ting immediately..

11

It looks like you have a trailing space in the datestamp field, which is not accounted for in the pattern. Correct the pattern used to extract this field and I do not see why it would not work.

1 Like
12

Thanks for your reply, but I try that with 1 more space between MMM and d and it didnt works.

date {
match => ["datestamp" , "EEE MMM d HH:mm:ss YYYY"]
target => "@timestamp"
}

13

I am referring to the space at the end off the string.

14

Okay , it didnt work but it dont said me _dateparsefailure

{
"path" => "/var/log/saprouterlog/saprouterlog.txt_a_20170510_060630-20170510_102448",
"datestamp" => "Wed May 10 06:12:51 2017 ",
"@timestamp" => 2017-05-10T04:12:51.000Z,
"@version" => "1",
"host" => "elkf",
"sap_session" => "C300/-",
"message" => "Wed May 10 06:12:51 2017 CONNECT FROM C300/- 172.16.58.199/55352",
"type" => "saplog",
"sap_action" => "CONNECT FROM ",
"sap_ipsource" => "172.16.58.199",
"tags" => [
[0] "saprouter1",
[1] "connect from"
],
"sap_port" => "/55352"

My filter date after add the space:

 date {
   match => ["datestamp","EEE MMM d HH:mm:ss YYYY "]
   target => "@timestamp"
 }

}
}

15

That seems to work. Remember that @timestamp is always in UTC. Instead of adding the space to the date filter pattern, I would recommend modifying the extraction pattern to get rid of it instead.

1 Like
16

Ah yes , i forgot it :slight_smile:
We can change UTC right ?

17

Timestamps in Elasticsearch are assumed to be in UTC, and a lot of functionality, including Kibana, relies on this. Trying to change this is therefore to ask for trouble.

1 Like
18

Ok, Thank you very much for your help Christian and thanks magnus for your great help :slight_smile:

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.