Weird? dateparse fail , if corrected, messages vanish


#1

Hello,

turns out that as long there is a dateparse fail, kibana will show the logfile entry based on @timestamp.
if there is no dateparse fail, kibana will show based on timestamp.

timestamp was filled all the time.

"messages are gone" and "vanished likely mean they are to be found at another time, because of not delivered timezone by switches

Hello folks, i am biting my teeth out with this.

as long as i have a non-matching date declaration, messages are found in kibana and get tagged with dateparsefailure . When i correct the date declarations, those messages are gone.

Any hints for that ?

Config with Dateparse fail:

filter {
if [type] == "cisco" {
grok {
patterns_dir => "/etc/logstash/patterns/"
match => ["message", "%{CISCO_IOS_HEADER} %{GREEDYDATA:cisco_message}"]
}
date {
match => ["timestamp",
"MMM dd HH:mm:ss",
"MMM d HH:mm:ss"
]
}
syslog_pri { }
grok {
patterns_dir => "/etc/logstash/patterns/"
match => [
"cisco_message", "%{CISCO_IOS_LINPROTO_UPDOWN}",
"cisco_message", "%{CISCO_IOS_ILPOWER}",
"cisco_message", "%{CISCO_IOS_IPACCESSLOGP}"
]
tag_on_failure => [ "_no_ciscomatch" ]
}
}
}


CISCO_IOS_HEADER <%{NUMBER:message_type_id}>%{NUMBER:internal_id}: %{GREEDYDATA}"%{HOSTNAME:hostname}:5005"]:frowning: %{INT:seq2_no}:)? %{CISCOTIMESTAMP:timestamp}: %?%{CISCO_IOS_TAG:ios_tag}:
CISCO_IOS_TAG %{WORD:ios_facility}-(%{WORD:ios_switchnumber}-)?%{INT:ios_severity}-%{WORD:ios_mnemonic}
CISCOTIMESTAMP %{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME}
CISCOTIMESTAMPTZ %{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME} %{TZ}

CISCO_IOS_LINPROTO_UPDOWN (Line protocol on Interface %{DATA:interface}, changed state to %{WORD:state})
CISCO_IOS_ILPOWER (Interface %{DATA:interface}: %{DATA:description}( (%{DATA:switch}))?)
CISCO_IOS_IPACCESSLOGP (fman_fp_image: list %{DATA:acl} permitted %{WORD:protocol} %{IP:src_ip}(%{INT:src_port}) -> %{IP:dst_ip}(%{INT:dst_port}), %{INT:count} packet)

AT (%{TIME} %{WORD} %{DAY} %{MONTH} %{MONTHDAY} %{YEAR})
UNKNOWN (unknown|Unknown|UNKNOWN)
UPDATE_TIMESTAMP (%{TIME} %{WORD} %{DAY} %{MONTH} %{MONTHDAY} %{YEAR})

Message there with dateparse fail:

<189>962319: swi-rha-edv4-nord: [syslog@9 s_id ="swi-rha-edv4-nord:5005"]: 962314: Jun 2 06:45:49.815: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/9: PD removed

Message without dateparse fail:

<190>5956344: [syslog@9 s_id ="ASR1:5005"]: Jun 2 09:52:35: %FMANFP-6-IPACCESSLOGP: F0: fman_fp_image: list ACL_SIT_IN permitted tcp 95.39.53.51(57648) -> 112.211.122.121(25), 1 packet

now, i "correct" the dateparsing declaration:

date {
match => ["timestamp",
"MMM dd HH:mm:ss.SSS",
"MMM d HH:mm:ss.SSS",
"MMM dd HH:mm:ss",
"MMM d HH:mm:ss"
]
}

no new messages from swi-rha-edv4-nord, no error log from logstash.

what i have seen, index declaration as viewed from kibana shows timestamp as string, not date

i am lost after hours of trying, hope someone got some pointers


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.