I've setup a filter in my logstash configuration and it doesn't match everytime the field has the correct value.
I am using Logstash 5.6.3 ,Elasticsearch 5.6.3 and Kibana 5.6.3
Filter:
filter {
if [program] == "RT_FLOW" {grok {
patterns_dir => ["/etc/logstash/patterns"]
match => { "message" => "%{RT_FLOW}"}
tag_on_failure => [_grokparsefailure]
}mutate {
add_tag => [ "Junos_parsed" ]
}
}
}
Message A (Doesn't get matched):
{
"_index": "logstash-2017.11.08",
"_type": "syslog",
"_id": "AV-aYxJyCrJhlYCvXZZp",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2017-11-08T14:40:34.000Z",
"@version": "1",
"host": "194.94.58.1",
"program": "RT_FLOW",
"message": "RT_FLOW_SESSION_CLOSE: session closed idle Timeout: XXX.XXX.XXX.XXX/49153->XXX.XXX.XXX.XXX/53 junos-dns-udp XXX.XXX.XXX.XXX/49153->XXX.XXX.XXX.XXX/53 None None 17 jh-allow-all XXX XXX 14663 2(114) 2(262) 4 UNKNOWN UNKNOWN N/A(N/A) fe-0/0/1.0 UNKNOWN",
"type": "syslog",
"logsource": "XXX"
},
"fields": {
"@timestamp": [
1510152034000
]
},
"sort": [
1510152034000
]
}
Message B (Gets parsed):
{
"_index": "firewall-2017.11",
"_type": "accounting_log",
"_id": "AV-cIFbtZ4vZR059cUPw",
"_version": 1,
"_score": null,
"_source": {
"src-ip": "XXX.XXX.XXX.XXX",
"dst-ip": "XXX.XXX.XXX.XXX",
"src-port": "1403",
"dst-port": "17600",
"program": "RT_FLOW",
"type": "syslog",
"to-zone": "junos-host",
"policy-name": "ssh-host",
"@version": "1",
"host": "XXX.XXX.XXX.XXX",
"elapsed-time": "4",
"event": "RT_FLOW_SESSION_CLOSE",
"close-reason": "session closed response received",
"nat-src-port": "1403",
"received": "0",
"message": "RT_FLOW_SESSION_CLOSE: session closed response received: XXX.XXX.XXX.XXX/1403->XXX.XXX.XXX.XXX/17600 icmp XXX.XXX.XXX.XXX/1403->XXX.XXX.XXX.XXX/17600 None None 1 ssh-host untrust junos-host 5808 1(84) 0(0) 4 UNKNOWN UNKNOWN N/A(N/A) fe-0/0/0.0 UNKNOWN",
"logsource": "XXXX",
"src-nat-rule-name": "None",
"sent": "84",
"tags": [
"Junos_parsed"
],
"nat-src-ip": "XXX.XXX.XXX.XXX",
"@timestamp": "2017-11-08T14:54:51.000Z",
"dst-nat-rule-name": "None",
"port": 39364,
"service": "icmp",
"nat-dst-ip": "XXX.XXX.XXX.XXX",
"protocol-id": "1",
"nat-dst-port": "17600",
"from-zone": "untrust",
"session-id": "5808"
},
"fields": {
"@timestamp": [
1510152891000
]
},
"sort": [
1510152891000
]
}
On another note: My Documents seem to be hanging behind about 10 min .
The "missing" 10 minutes get filled as the time progresses
Monitoring of my Logstash :
Thanks in advance 