"_dateparsefailure" due to Timestamp Cisco IOS

I used the conf file from here and the pattern below in a pattern file referenced by the conf file.

.conf grok section

grok {
  # There are a couple of custom patterns associated with this filter.
  patterns_dir => [ "/etc/logstash/conf.d/patterns/ios" ]

  match => [
    # IOS
    "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence})?:( %{HOSTNAME:hostname}:)? (%{INT:cisco_seq_num}: )?.?%{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: ( )?%{GREEDYDATA:message}",
    "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence})?:( %{HOSTNAME:hostname}:)? (%{INT:cisco_seq_num}: )?.?%{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: ( )?%{GREEDYDATA:message}"
  ]

  overwrite => [ "message" ]

  add_tag => [ "ciscoios" ]

  remove_field => [ "syslog5424_pri", "@version" ]
}
}

pattern file /etc/logstash/conf.d/patterns/ios/patterns-ios:

CISCOTIMESTAMPTZ %{CISCOTIMESTAMP}( %{TZ})?

the %{TZ} pattern is in the default grok patterns for logstash as:

TZ (?:[APMCE][SD]T|UTC)

I'm receiving "_dateparsefailure" ONLY for logs that specify a timestamp OTHER THAN "EST" e.g. CST|PST.

If I change the clock on the Cisco device to use EST rather than PST|CST the "_dateparsefailure" stops.

You have to share how your input log looks, then someone will be able to help you

Ah of course, thanks!

The parse fail happens for only the PST and CST timestamped logs.

<189>261731: HOSTNAME1: Nov 27 08:00:00.838 PST: %SYS-5-CONFIG_I: Configured from console by username on line# (XX.XX.XX.XX)
<189>14022: HOSTNAME2: Nov 27 09:58:01.178 CST: %SYS-5-CONFIG_I: Configured from console by username on line# (XX.XX.XX.XX)
<189>183040: HOSTNAME3: 183040: Nov 27 10:45:10.946 EST: %SYS-5-CONFIG_I: Configured from console by username on line# (XX.XX.XX.XX)

Parse as such below.

{
                 "host" => "XX.XX.XX.XX",
          "fingerprint" => "8b13ea19088714a1f29feb6201397e76436ed2e8",
             "hostname" => "HOSTNAME1",
             "facility" => "SYS",
         "log_sequence" => "261735",
             "log_date" => "Nov 27 08:02:21.024 PST",
        "facility_full" => "Operating system",
           "@timestamp" => 2019-11-27T16:02:20.444Z,
                 "tags" => [
        [0] "ciscoios",
        [1] "_dateparsefailure"
    ],
                 "type" => "syslog-cisco",
    "facility_mnemonic" => "CONFIG_I",
              "message" => "Configured from console by username on line# (XX.XX.XX.XX)",
       "severity_level" => "5 - Notification"
}

and

{
              "message" => "Configured from console by username on line# (XX.XX.XX.XX)",
           "@timestamp" => 2019-11-27T15:57:06.890Z,
        "facility_full" => "Operating system",
             "facility" => "SYS",
                 "host" => "XX.XX.XX.XX",
                 "tags" => [
        [0] "ciscoios",
        [1] "_dateparsefailure"
    ],
    "facility_mnemonic" => "CONFIG_I",
       "severity_level" => "5 - Notification",
             "hostname" => "HOSTNAME2",
                 "type" => "syslog-cisco",
          "fingerprint" => "78c95a9ccebe1fa19d14241c251ac7d28381b2eb",
         "log_sequence" => "14021",
             "log_date" => "Nov 27 09:57:05.879 CST"
}

and

{
              "message" => "Configured from console by username on line# (XX.XX.XX.XX)",
           "@timestamp" => 2019-11-27T15:54:03.939Z,
        "facility_full" => "Operating system",
        "cisco_seq_num" => "183041",
             "facility" => "SYS",
                 "host" => "XX.XX.XX.XX",
                 "tags" => [
        [0] "ciscoios"
    ],
    "facility_mnemonic" => "CONFIG_I",
       "severity_level" => "5 - Notification",
             "hostname" => "HOSTNAME3",
                 "type" => "syslog-cisco",
          "fingerprint" => "8f01d06ce0e47565930c10bd7c468bf7c0021c8e",
         "log_sequence" => "183041",
             "log_date" => "Nov 27 10:54:03.939 EST"

Nov 27 08:00:00.838 PST(or any other EST)
%{CISCOTIMESTAMP:log_date} %{GREEDYDATA:timestamp}

Try the above in grok debugger , and mutate your whole filter accordingly.Go step by step while building grok.

OK, so the grok works just fine, the issue is that you are getting a _dateparsefailure when you pass [log_date] to a date filter. That happens because PST and CST are ambiguous. There is no way for the date filter to know what PST refers to. You can use mutate+gsub to disambiguate it by replacing PST with US/Pacific, or PST8PDT, or Pacific/Pitcairn depending on which one you are using.

Yea I just noticed that in the logstash docs referencing the joda timzones. I'll try your solution

Thanks!

I used this below and it worked perfectly thanks!

mutate{
                gsub => [
               "log_date", "EST|EDT", "EST5EDT",
               "log_date", "CST|CDT", "CST6CDT",
               "log_date", "MST|MDT", "MST7MDT",
               "log_date", "PST|PDT", "PST8PDT"
        ]
        }