I have filebeat pushing data to logstash. Logstash is only able to parse the first entry pushed by beats and the rest are not getting filtered.
This is the record present in log file:
2020-07-29 08:35:18,042 INFO [gov.nist.javax.sip.stack.SIPTransactionStack] (CTSI-UDPMessageChannelThread-31) <message
from="172.16.0.43:8993"
to="172.16.0.43:9990"
time="1595234178042"
isSender="false"
transactionId="z9hg4bk894d.3f25ad91.0"
callId="NS_764283@216.59.58.221"
firstLine="OPTIONS sip:pctnxtgn.mvoipctsi.com:8993 SIP/2.0"
>
<![CDATA[OPTIONS sip:pctnxtgn.mvoipctsi.com:8993 SIP/2.0
Record-Route: <sip:172.16.0.43:8993;r2=on;lr>
Record-Route: <sip:209.15.246.95:8993;r2=on;lr>
Via: SIP/2.0/UDP 172.16.0.43:8993;branch=z9hG4bK894d.3f25ad91.0
Via: SIP/2.0/UDP 216.59.58.221:11640;received=216.59.58.221;rport=11640;branch=z9hG4bK2DD70764283
Call-ID: NS_764283@216.59.58.221
To: <sip:NSMON@backend>
From: <sip:NSMON@netscaler>
CSeq: 1 OPTIONS
Subject: monitoring
Max-Forwards: 1
User-Agent: netscaler/7.0
Contact: "NSMON" <sip:NSMON@backend;transport=UDP>;expires=3600
X-Source-URI: sip:216.59.58.221:11640;transport=udp
Content-Length: 0
]]>
</message>
This is the message sent by FileBeat:
"message": "2020-07-29 08:07:18,042 INFO [gov.nist.javax.sip.stack.SIPTransactionStack] (CTSI-UDPMessageChannelThread-31) <message\nfrom=\"172.16.0.43:9990\" \nto=\"172.16.0.43:8993\" \ntime=\"1595234178042\"\nisSender=\"true\" \ntransactionId=\"z9hg4bk894d.3f25ad91.0\" \ncallId=\"NS_764283@216.59.58.221\" \nfirstLine=\"SIP/2.0 483 Too many hops\" \n>\n<![CDATA[SIP/2.0 483 Too many hops\nCSeq: 1 OPTIONS\nCall-ID: NS_764283@216.59.58.221\nFrom: <sip:NSMON@netscaler>\nTo: <sip:NSMON@backend>\nVia: SIP/2.0/UDP 172.16.0.43:8993;branch=z9hG4bK894d.3f25ad91.0\nVia: SIP/2.0/UDP 216.59.58.221:11640;received=216.59.58.221;rport=11640;branch=z9hG4bK2DD70764283\nContent-Length: 0\n\n]]>"
My Logstash filter is as follows:
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_time}%{SPACE}%{LOGLEVEL:log_level}%{SPACE}\[%{JAVAFILE:className}\]%{SPACE}\(%{NOTSPACE:log_thread}\).*from\=\"%{IP:from_ip}:%{POSINT:fro$
}
mutate {
gsub => [
# replace all commas with dots
"log_time", ",", "."
]
}
mutate {
gsub => [
"log_time", " ", ";"
]
}
date {
locale => "en"
match => ["log_time", "YYYY-MM-dd;HH:mm:ss.SSS"]
timezone => "UTC"
target => "log_time"
}
}
In Kibana I am able to see only one log entry. I am clueless as to what is happening. Can you guys give me any advice?