Log files are not current

Elastic Stack Logstash
#1

I am running the Kibana GUI and the log files are not current: This the output of the first log:

March 24th 2016, 15:17:08.747
message:2015-08-13 02:36:24,038 DEBUG [AptService] [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] Request APT :: Request ID : 157471, Priority

The time stamp is : March 24th 2016, 15:17:08.747
the message date is 8/13/2015. How come the log file doesn't generate the most current logs?
How can I correct this.

#2

It seems like you don't have a date filter to parse the date in the input logs.

#3

How can I do that?

#4

You should use a date filter, in your example:
Timestamp is: March 24th 2016, 15:17:08.747
i used this: joda.timeFormat
Snippet below should works :wink:

filter{
date { match => ["MMMM dddd YYYY, HH:mm:ss.SSS"] }
}
#5

Sorry I am new to ELK.

I have 4 files /etc/logstash/conf.d
rw-r--r--. 1 root root 194 Mar 10 13:32 02-beats-input.conf
-rw-r--r--. 1 root root 155 Mar 22 10:15 10-syslog.conf
-rw-r--r--. 1 root root 470 Mar 24 20:24 10-syslog-filter.conf
-rw-r--r--. 1 root root 220 Mar 23 14:26 30-elasticsearch-output.conf

Do I put the code in here, this the contents of file 10-syslog-filter.conf, where would I put it exactly.

filter {

if [type] == "syslog" {

grok {

  match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

  add_field => [ "received_at", "%{@timestamp}" ]

  add_field => [ "received_from", "%{host}" ]

}

syslog_pri { }

date {

  match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

}

}

}

#6

The field name is missing. Should be something like:

date { match => ["name-of-field-with-timestamp", "MMMM dddd YYYY, HH:mm:ss.SSS"] }
#7

That date filter is for syslog messages but the message with the incorrect timestamp doesn't look like a syslog message. It probably came via the beats input, and those message are apparently not processed by a working date filter.

#8

so where would I put the date field?

#9

Put it in whatever configuration file you have for parsing the events that arrive via the beats input. If you're not parsing those events you should start doing that. Since you have a separate file for taking care of syslog events I suggest you create a similar one for dealing with events arriving via the beats input.

#10