Syslog date without year

Elastic Stack Logstash
1

Hi,

I'm using Logstash to analyze my syslog files. My syslog files have the date like this: Jun 30 06:34:35.
I want have the year to, because I need to use date to filter visualizations on kibana.

I don't use kibana @timestamp because the syslog data are upload just when I need and @timestamp it's different of each syslog date.

How can I do this?

2

Sorry, I don't understand why you can't use @timestamp. The date filter will use the year when the Logstash process started as the default year when date being parsed doesn't contain a year. Are you saying that you're not pushing logs to Elasticsearch continuously but import data in batches as needed?

3

Exactly, I don't use continualy but I push the files when I need. Because of this I don't use @timestamp.

4

Okay, but that doesn't make sense. With Logstash one normally uses the date filter to populate the @timestamp field with the time that the event occurred.

Either way having a logfile without year information is problematic, especially when you batch-process old files. Assuming you process log files within a year, perhaps you can use a ruby filter to compare the current time with the month and day from the log entry and choose the correct year based on that.

5

OK, i will try, thanks for help me!

Best regards.

6

How can I convert string to date?

7

Use the date filter. If you're writing Ruby code for a ruby filter you'll have to look into Ruby's date and time functions.

8

Ok I will try, thanks for help again.

9

I am following this thread : http://stackoverflow.com/questions/29384754/syslog-timestamp-without-year

My requirement is :
If syslog message comes with year in timestamp, I should match in date filter and take that year as time and put it in elasticsearch, If there is not year in syslog timestamp, then I need to have some default year / some default value which I will know in future, How can I do that ?

My date filter is :
date {
target => "@timestamp"
match => [ "syslog_timestamp",
"MMM d HH:mm:ss YYYY",
"MMM dd HH:mm:ss YYYY" ]
timezone => "UTC"
}

Here If syslog_timestamp does not contains year then this fails.

10

Any reply on this one ?

11

Hi,
In your message if you don't have year, you can force a year.

E.g.:

filter {
mutate {
replace => ["timestamp", "%{timestamp} 2014"]
}
date {
locale => "en"
match => ["timestamp",
"MMM d HH:mm:ss YYYY",
"MMM dd HH:mm:ss YYYY",
"ISO8601"]
}
}

PS: Magnus Black answer in: http://stackoverflow.com/questions/29384754/syslog-timestamp-without-year