Logstash not ignoring syslog event date

Elastic Stack Logstash
1

I have the following setup

ELK server on Ubuntu 14.04 - timezone: EDT
syslogd server on Ubuntu 10.04 - timezone: UTC (logstash-forwarder installed)
app server 1 on Ubuntu 14.04 - timezone: EST (sending logs to syslogd server)
app server 2 on Ubuntu 14.04 - timezone: UTC (sending logs to syslogd server)

When my app server 1 sends logs I see the logs instantly in Kibana
When my app server 2 sends logs I don't see the logs in Kibana.

app server1 event logged in logstash.stdout - "@timestamp" => "2015-10-30T00:59:48.000Z",
app server2 event logged in logstash.stdout - "@timestamp" => "2015-10-30T05:00:19.000Z",

Looks like my server2 event timestamp is being influenced by the UTC timezone. here's my syslog conf. What am i missing?

My 10-syslog.conf file

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
  }
}
2

What was the actual time when server 2 produced the timestamp above ("2015-10-30T05:00:19.000Z")? If you don't know, capture some live messages emitted by Logstash on server 2.

I assume you have a date filter to parse syslog_timestamp into @timestamp.

3

Hello Magnus,

Thank you for the reply, I think i got it. I was backing up older conf files in same directory as i made changes and apparently logstash continues to read those although they don't have the .conf extension

  • 10-syslog.conf.old
  • 10-syslog.conf.orig

Discovered it when i started seeing syslog_hostname have 3 duplicates. and ofcourse my .orig still has the date filter.

Thank you