@timestamp is not matching the actual event time from the router syslog

Elastic Stack Logstash
1

elasticsearch is creating indexes with a future date. @timestamp is not matching the actual event time from the router syslog . The actual event time is Nov 30 2018. Could anyone tell me why the index and @timestamp is not 2018 ?

"Nov 30 10:40:16 system-aaaaaa 172: *Nov 30 01:49:00.001: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:router bgp 65008"

"filebeat-6.2.4-2019.11.30"

content of one of the documents

  {
    "_index": "filebeat-6.2.4-2019.11.30",
    "_type": "doc",
    "_id": "_sZBJ2gBAn3SQNgAVupJ",
    "_score": 1,
    "_source": {
      "offset": 56292,
      "beat": {
        "hostname": "xxx-xxxx-02",
        "version": "6.2.4",
        "name": "xxx-xxxx-02"
      },
      "@timestamp": "2019-11-30T02:40:16.000Z",
      "tags": [
        "beats_input_codec_plain_applied"
      ],
      "prospector": {
        "type": "log"
      },
      "source": "/var/log/HOSTS/system-aaaaaa.log-20190106",
      "system": {
        "syslog": {
          "hostname": "system-aaaaaa",
          "message": "*Nov 30 01:49:00.001: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:router bgp 65008 ",
          "program": "172",
          "timestamp": "Nov 30 10:40:16"
        }
      },
      "host": "xxx-xxxx-02",
      "@version": "1",
      "fileset": {
        "module": "system",
        "name": "syslog"
      }
    }

this is my logstash configuration.

input {
beats {
port => 5044
host => "0.0.0.0"
}
}
filter {
if [fileset][module] == "system" {
if [fileset][name] == "syslog" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
remove_field => "message"
}
date {
match => [ "[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
}
output {
elasticsearch {
user => logstash_system
password => xxxxxxx
hosts => ["es-host:3306"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

2

Because Logstash assumes that the missing year in the timestamp is this year.

Not sure how to resolve that though sorry.

3

If you assume the date can not be in the future, you may able to do something like this:

input {
  generator {
    lines => ['Jan 01 01:49:00.001',
              'Nov 30 01:49:00.001']
    count => 1
  } 
} 

filter {
  date {
    match => [ "message", "MMM d HH:mm:ss.SSS", "MMM dd HH:mm:ss.SSS" ]
    target=> "[@metadata][tmpdate]"
  }

  if [@metadata][tmpdate] > [@timestamp] {
    mutate {
      add_field => { "[@metadata][fulltmpdate]" => "2018 %{[message}"}
    }

    date {
      match => [ "[@metadata][fulltmpdate]", "yyyy MMM d HH:mm:ss.SSS", "yyyy MMM dd HH:mm:ss.SSS" ]
    }
  } else {
    date {
      match => [ "message", "MMM d HH:mm:ss.SSS", "MMM dd HH:mm:ss.SSS" ]
    }
  }
}

output {
  stdout { codec => rubydebug { metadata => true} }
}

Adapt the fields to fit your data structure.

1 Like
4

but according to the change log, as of version 2.1.0 the year rollover is handled internally. i should not be seeing this behaviour. can you clarify ?

/usr/share/logstash/bin/logstash-plugin list --verbose |grep input

logstash-filter-date (3.1.9)

5

It might be a regression then, can you raise an issue under the plugin so we can dig into it?

6

made the changes to the logstash configurations per recommendation from Christian. I will monitor it for sometime. thanks

7

The change you pointed to only seems to handle the rollover between December and January. As your data belongs to November and comes over a month late, it is not caught by this fix.

8

Christian, are you talking about how logstash handles the rollover ?

i am monitoring the data after apply the recommendations to the logstash configuration . seems to be working fine. i can see all date from 2018 with the correct timestamp now.

what happens when we roll over to 2020 ? are we going to face a similar problem ?

9

If data does not arrive delayed more than a month I suspect the fix you linked to will handle it.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.