Another @timestamp question... non-standard timestamp transformation

Elastic Stack Logstash
1

Hi there, I'm new to all of this.

I'm using Logstash with Elastic Search and finding the @timestamp transformation frustrating for my non-standard log format. Any help you can provide me would be appreciated.

Here is a log sample:

No.,Record,Date,Time,Source,Site,+/-,Event,Mode,User,Details
2595,2543,01Jul19,00:00:00,TLA,607,,,,,message long text
2655,2603,01Jul19,00:00:00,BLE,3392,,,,,another long open text structure

Here is the filter I am using:

filter {
grok {
match => {"message" => "%{NUMBER:line},%{NUMBER:record},%{MONTHDAY:day}%{NOTSPACE:month}%{YEAR:year},%{TIME:timeofday},%{WORD:region},%{NUMBER:site},.*,%{GREEDYDATA:message}"}
}
date {
match => ["replace_timestamp", "ddMMMyy,kk:mm:ss"]
target => "@timestamp"
}
}

I am having a lot of trouble getting the timestamp to work... can anyone see the problem i'm having?
Thanks for your patience.

2

I would suggest using a csv filter to parse the input lines.

What creates the [replace_timestamp] field?

1 Like
3

I have no idea about the replace_timestamp field - i was following some of the other posts here and I don't really understand and I can't find the documentation very thorough for me.

I have no idea If i can use the multiple items i've already grokked to do this?

CSV filter.... ok i'll look that up. Thank you.

New config:

filter {
csv {
separator => ","
columns => ["line","record","date","time","source","site","+/-","event","mode","user","details"]
add_field => { "replace_timestamp" => "%{date} %{time}" }
}
date {
match => ["replace_timestamp", "ddMMMyy kk:mm:ss"]
target => "@timestamp"
}
}

Now the output looks like this:

{
"site" => "328",
"@timestamp" => 2019-08-28T00:37:51.491Z,
"mode" => "Isol",
> "tags" => [
> [0] "_dateparsefailure"
> ],
"message" => "38415,25499,01Jul19,00:04:06,WOL,328,+,LM,Isol,,TEXT",
"replace_timestamp" => "01Jul19 00:04:06",
"+/-" => "+",
"time" => "00:04:06",
"details" => "TEXT",
"line" => "38415",
"user" => nil,
"source" => "WOL",
"event" => "LM",
"@version" => "1",
"host" => "elasticsearch",
"record" => "25499",
"date" => "01Jul19",
"path" => "/home/elasticsearch/eventlogs/sm_WOL_July.csv"
}

Looks like it can create the replace timestamp field now, but can't parse it?
Thank you!

4

You have used kk for hour, but your hour is 00, so clearly you need HH (which goes from 00 to 23), not kk (which goes from 01 to 24).

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.