Unable to read parse log entries in Logstash

I have filebeat pushing data to logstash. Logstash is only able to parse the first entry pushed by beats and the rest are not getting filtered.

This is the record present in log file:

2020-07-29 08:35:18,042 INFO  [gov.nist.javax.sip.stack.SIPTransactionStack] (CTSI-UDPMessageChannelThread-31) <message
from="172.16.0.43:8993" 
to="172.16.0.43:9990" 
time="1595234178042"
isSender="false" 
transactionId="z9hg4bk894d.3f25ad91.0" 
callId="NS_764283@216.59.58.221" 
firstLine="OPTIONS sip:pctnxtgn.mvoipctsi.com:8993 SIP/2.0" 
>
<![CDATA[OPTIONS sip:pctnxtgn.mvoipctsi.com:8993 SIP/2.0
Record-Route: <sip:172.16.0.43:8993;r2=on;lr>
Record-Route: <sip:209.15.246.95:8993;r2=on;lr>
Via: SIP/2.0/UDP 172.16.0.43:8993;branch=z9hG4bK894d.3f25ad91.0
Via: SIP/2.0/UDP 216.59.58.221:11640;received=216.59.58.221;rport=11640;branch=z9hG4bK2DD70764283
Call-ID: NS_764283@216.59.58.221
To: <sip:NSMON@backend>
From: <sip:NSMON@netscaler>
CSeq: 1 OPTIONS
Subject: monitoring
Max-Forwards: 1
User-Agent: netscaler/7.0
Contact: "NSMON" <sip:NSMON@backend;transport=UDP>;expires=3600
X-Source-URI: sip:216.59.58.221:11640;transport=udp
Content-Length: 0

]]>
</message>

This is the message sent by FileBeat:

"message": "2020-07-29 08:07:18,042 INFO [gov.nist.javax.sip.stack.SIPTransactionStack] (CTSI-UDPMessageChannelThread-31) <message\nfrom=\"172.16.0.43:9990\" \nto=\"172.16.0.43:8993\" \ntime=\"1595234178042\"\nisSender=\"true\" \ntransactionId=\"z9hg4bk894d.3f25ad91.0\" \ncallId=\"NS_764283@216.59.58.221\" \nfirstLine=\"SIP/2.0 483 Too many hops\" \n>\n<![CDATA[SIP/2.0 483 Too many hops\nCSeq: 1 OPTIONS\nCall-ID: NS_764283@216.59.58.221\nFrom: <sip:NSMON@netscaler>\nTo: <sip:NSMON@backend>\nVia: SIP/2.0/UDP 172.16.0.43:8993;branch=z9hG4bK894d.3f25ad91.0\nVia: SIP/2.0/UDP 216.59.58.221:11640;received=216.59.58.221;rport=11640;branch=z9hG4bK2DD70764283\nContent-Length: 0\n\n]]>"

My Logstash filter is as follows:

filter {
    grok {
      match => ["message", "%{TIMESTAMP_ISO8601:log_time}%{SPACE}%{LOGLEVEL:log_level}%{SPACE}\[%{JAVAFILE:className}\]%{SPACE}\(%{NOTSPACE:log_thread}\).*from\=\"%{IP:from_ip}:%{POSINT:fro$
    }

        mutate {
            gsub => [
                # replace all commas with dots
                "log_time", ",", "."
                ]
        }

        mutate {
            gsub => [
                "log_time", " ", ";"
                ]
        }

        date {
            locale => "en"
            match => ["log_time", "YYYY-MM-dd;HH:mm:ss.SSS"]
            timezone => "UTC"
            target => "log_time"
        }
}

In Kibana I am able to see only one log entry. I am clueless as to what is happening. Can you guys give me any advice?

what's your filebeat input setting? I feel the multi-line is causing problem at filebeat side.
Try something like below in inputs of your filebeat if you using file input type

        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601}"
            negate => "true"
            what => "previous"
        }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.