Grok TIME

Elastic Stack Logstash
#1

Hi All ,

I am new to ELK and trying to forward a non-real time log by applying grok in config file to pick the log time .I referred few previous discussions to find that Grok could be used to parse the data & Date{} could be used to overwrite with log time.
PFB the details of log & config file.
Please help me where I am missing .
Input Data-
A824209B7C244E64A1370281C2FF 3E975EA9197F4A91BF3869F14D67 ReadyForProcessing 12:30:14 AM

Config File:
input {
file {
path => "Location\Sample.csv"

      start_position => "beginning"
  }

}
output {
elasticsearch {
action => "index"
index => "newindex"
}
}

filter {
csv {
columns => ["ConversationID", "Conversation_StateID", "Conversation_Status_Desc", "ConversationStateComments","CreationTime"]
separator => ","
}
grok {

        match => { "message" => "%{TIME:timestamp:date}" }
        }
	date {
        match => [ "timestamp", "HH:mm:ss" ]
        }

}

Please help .

Thanks ,
Prateek

#2

Okay, I see the time at the end of the input string. But where does the date come from? Do you want to use today's date or what?

#3

The date when the log is created ,its not a current date.
A824209B7C244E64A1370281C2FF 3E975EA9197F4A91BF3869F14D67 ReadyForProcessing 2016-02-25 12:30:14 AM

#4

A824209B7C244E64A1370281C2FF 3E975EA9197F4A91BF3869F14D67 ReadyForProcessing 2016-02-25 12:30:14 AM

So... the date is in the log line after all?

#5

Yes the date comes in a log line .

#6

Okay, so what's the problem? You have the date in one field and the time in another?

#7

No the date comes in same log line with time in a format YYYY-MM-DD HH:MM:SS

#8

Yes, that's understood but what I don't understand is what you're asking. You're using the csv filter to parse the data so you should get the date in one field and the time in another?

#9

Yes I am using csv filter to parse the data.
I want to get a time associated with logs primarily but through csv filter it is coming up in a string format & analysed field.
@timestamp is picking the current system's date and time so not able to correlate the data with log time.

#10

You're already attempting to use the date filter and that's indeed the correct approach. That filter requires the date and the time to be in one field. See http://stackoverflow.com/a/28882096/414355 if you have the date and the time in different fields and need to combine them.

If you get stuck again and need further help you need to provide hard facts; the exact configuration, the exact input, the exact output you get (use a stdout { codec => rubydebug } output, not elasticsearch), and what you would like to get.

#11

Thank you for your patience and help.
As per suggested I combined the fields in data file itself.
The slight change done for grok in config file(tried & verified first on grok debugger:-http://grokdebug.herokuapp.com/)

Filter Part of config file -
filter {
csv {
columns => ["ConversationID", "Conversation_StateID", "Conversation_Status_Desc", "ConversationStateComments","CreationTime"]
separator => ","
}
grok {

        match => { "message" => "%{DATESTAMP:timestamp:date}" }
        }
	date {
        match => [ "timestamp", "YY-MM-DD HH:mm:ss" ]
        }
#12