Another @timestamp question... non-standard timestamp transformation

[Kalydrae (S Walker)]

Hi there, I'm new to all of this.

I'm using Logstash with Elastic Search and finding the @timestamp transformation frustrating for my non-standard log format. Any help you can provide me would be appreciated.

Here is a log sample:

No.,Record,Date,Time,Source,Site,+/-,Event,Mode,User,Details
2595,2543,01Jul19,00:00:00,TLA,607,,,,,message long text
2655,2603,01Jul19,00:00:00,BLE,3392,,,,,another long open text structure

Here is the filter I am using:

filter {
grok {
match => {"message" => "%{NUMBER:line},%{NUMBER:record},%{MONTHDAY:day}%{NOTSPACE:month}%{YEAR:year},%{TIME:timeofday},%{WORD:region},%{NUMBER:site},.*,%{GREEDYDATA:message}"}
}
date {
match => ["replace_timestamp", "ddMMMyy,kk:mm:ss"]
target => "@timestamp"
}
}

I am having a lot of trouble getting the timestamp to work... can anyone see the problem i'm having?
Thanks for your patience.

[Badger]

I would suggest using a csv filter to parse the input lines.

What creates the [replace_timestamp] field?

[Kalydrae (S Walker)]

I have no idea about the replace_timestamp field - i was following some of the other posts here and I don't really understand and I can't find the documentation very thorough for me.

Looking for idea to preprocess logs

Instead of pre-order the data, use the timestamp from the event and set that as the timestamp to index in elasticsearch. Then it nicely sorted in kibana when you view the data.. You can use a filter like this (this does not match your timestamp, it is just an example). filter { grok { match => { "message" => "%{TIMESTAMP_ISO8601:replace_timestamp}" } } date { match => ['replace_timestamp', 'yyyy-MM-dd HH:mm:ss'] timezone => "UTC" target => "@timestamp" …

I have no idea If i can use the multiple items i've already grokked to do this?

CSV filter.... ok i'll look that up. Thank you.

New config:

filter {
csv {
separator => ","
columns => ["line","record","date","time","source","site","+/-","event","mode","user","details"]
add_field => { "replace_timestamp" => "%{date} %{time}" }
}
date {
match => ["replace_timestamp", "ddMMMyy kk:mm:ss"]
target => "@timestamp"
}
}

Now the output looks like this:

{
"site" => "328",
"@timestamp" => 2019-08-28T00:37:51.491Z,
"mode" => "Isol",
> "tags" => [
> [0] "_dateparsefailure"
> ],
"message" => "38415,25499,01Jul19,00:04:06,WOL,328,+,LM,Isol,,TEXT",
"replace_timestamp" => "01Jul19 00:04:06",
"+/-" => "+",
"time" => "00:04:06",
"details" => "TEXT",
"line" => "38415",
"user" => nil,
"source" => "WOL",
"event" => "LM",
"@version" => "1",
"host" => "elasticsearch",
"record" => "25499",
"date" => "01Jul19",
"path" => "/home/elasticsearch/eventlogs/sm_WOL_July.csv"
}

Looks like it can create the replace timestamp field now, but can't parse it?
Thank you!

[Badger]

You have used kk for hour, but your hour is 00, so clearly you need HH (which goes from 00 to 23), not kk (which goes from 01 to 24).