Problem with timestamp

Elastic Stack Logstash
1

hi i am using following filter to overwrite the @timestamp, but its not working. may i know the issue.

filter{
date{
match =>["timeStamp", "EEE yyyy MMM dd HH:mm:ss:SSS"]
remove_field => ["timeStamp"]
}
}

2

What does the input timestamp look like? What does the resulting event look like? What's in the Logstash logs (the date filter logs information about all parse failures)?

3

Hi magnus,

following is the output from logstash

{"ipaddress":"169.254.133.10","secEventType":"authentication","messageID":"messageID0","userName":"u
serName0","Info":"Info0","tags":["_dateparsefailure"],"timeStamp":"Tue 2017 Jan 24, 02:36:31:088","p
ath":"C:/satish/SecurityEvent-2017-Jan-24.log","@timestamp":"2017-01-24T02:54:50.239Z","info1":"abc0
","@version":"1","host":"L24660WIN","ID":"000891b7-1907-40ed-a1d5-d0a51be07ad3","info2":"xyz0"}

In the logstash config i have given the time format in the date filter, so as to convert it in to date type
"timeStamp":"Tue 2017 Jan 24, 02:36:31:088" In GMT format(this is logevent time)
"@timestamp":"2017-01-24T02:54:50.239Z"
not matching my time format.

also i want to remove timestamp which is of string type, that is also not working.

4

What's in the Logstash logs (the date filter logs information about all parse failures)?

5

logstash log is same as posted in following thread

6

logstash log is same as posted in following thread

Okay, but that log doesn't contain anything from the date filter. I'm quite sure there's something in there. If you temporarily replace your elasticsearch output with stdout { codec => rubydebug } output this problem will be easier to debug.

7

Hi magnus,

i found the issue it is with the "," missed in the timeformat. This got fixed.
i had another issue

my input log to logstash is
{"ID":"c0306973-4c74-4cf0-9026-974b09dbab28","secEventType":"Authentication","timeStamp":"Tue 2017 Jan 24, 10:22:47:888","userName":"userName0","ipaddress":"169.254.133.10","Info":"Info0","messageID":"messageID0","info1":"abc0","info2":"xyz0"}

out put is
{"ipaddress":"169.254.133.10","secEventType":"authentication","messageID":"messageID0","userName":"u
serName0","Info":"Info0","path":"C:/satish/SecurityEvent-2017-Jan-24.log","@timestamp":"2017-01-24T0
4:52:47.888Z","info1":"abc0","@version":"1","host":"L24660WIN","ID":"c0306973-4c74-4cf0-9026-974b09d
bab28","info2":"xyz0"}

why the time format is not persisting? Also is it possible to keep timestamp(my field) and avoid creation of @timestamp(created by logstash)?

8

why the time format is not persisting?

The purpose of the date filter is to parse a timestamp and store it in a field in a canonical format that's recognized by (for example) Elasticsearch. If you don't want to change the time format then you shouldn't use the date filter but then you'll instead have to arrange for Elasticsearch to do the parsing so that documents get the correct timestamp.

Also is it possible to keep timestamp(my field) and avoid creation of @timestamp(created by logstash)?

Look at the filter's target option.

9

thanks for the suggestion, it worked

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.