Need Help in Logstash Performance Issue

Elastic Stack Logstash
1

Hi All,

I am having a requirement to filter some 50 GB data thru logstash to view the real time data in kibana

currently there is a slight lag in logstash processing could you pl let know what all the possible ways to increase the speed

while ingesting, logstash took 25mts to process 1.5GB of data

Thanks
Arun

2

What version?
What is your config?
What OS?
What JVM?
What hardware does the instance run on?

3

Logstash 5.6.3
ubuntu16.04
JVM 1.8.0_144
4core 8GB RAM

config - input {
file {
path => "/var/log/XS.txt"
start_position => beginning
codec => multiline {
pattern => "^(?[0-9]+.[0-9]+.[0-9]+) %{TIME:date} %{WORD:TimeType} "
negate => true
what => previous
}
}
}

filter {
if ["Sip |", " INVITE"] in [message] or ["SipMedia |", " INVITE"] in [message] {
if "SIP/2.0 " in [message]
{
if " IN " in [message]
{
dissect {
mapping => {
message => "%{+Timestamp} %{+Timestamp} %{}Bytes %{Direction:} %{}SIP/2.0 %{Sip_Response} %{}<sip:%{SipToUser:}@%{}<sip:%{SipFromUser:}@%{}CSeq:%{} %{Response_For_Msg:}Content-Length:%{}via:%{} %{ServerName:};%{}"
}
}
}
else if " OUT " in [message]
{
dissect {
mapping => {
message => "%{+Timestamp} %{+Timestamp} %{}Bytes %{Direction:} %{}SIP/2.0 %{Sip_Response} %{}<sip:%
{SipFromUser:}@%{}<sip:%{SipToUser:}@%{}CSeq:%{} %{Response_For_Msg:}:%{}"
}
}
}
else
{ drop{} }
}
else if " IN " in [message]
{
dissect {
mapping => {
message => "%{+Timestamp} %{+Timestamp} %{}Bytes %{Direction:} %{}<sip:%{SipToUser:}@%{}<sip:%{SipFromUser:}@%{}CSeq:%{} %{Response_For_Msg:}:%{}via:%{} %{ServerName:};%{}"
}
}
}
else if " OUT " in [message]
{
dissect {
mapping => {
message => "%{+Timestamp} %{+Timestamp} %{}Bytes %{Direction:} %{}<sip:%{SipFromUser:}@%{}<sip:%{SipToUser:}@%{}CSeq:%{} %{Response_For_Msg:}:%{}"
}
}
}

           else{
              drop{}
           }
        }
     }
4

Please format your code using the </> button, or markdown style back ticks, it's really hard to read as is :slight_smile:

5 
input {
    file {
        path => "/var/log/XS.txt"
        start_position => beginning
        sincedb_path => "/dev/null"
        codec => multiline {
        pattern => "^(?<timestamp>[0-9]+.[0-9]+.[0-9]+) %{TIME:date} %{WORD:TimeType} "
        negate => true
        what => previous
        }
    }
}

filter {
            if ["Sip |", " INVITE"] in [message] or ["SipMedia |", " INVITE"]  in [message] {
               if "SIP/2.0 " in [message]
               {
                   if " IN " in [message]
                   {
                      dissect {
                        mapping => {
                         message => "%{+Timestamp} %{+Timestamp} %{}Bytes %{Direction:} %{}SIP/2.0  %{Sip_Response} %{}<sip:%{SipToUser:}@%{}<sip:%{SipFromUser:}@%{}CSeq:%{} %{Response_For_Msg:}Content-Length:%{}via:%{} %{ServerName:};%{}"
                         }
                      }
                   }
                   else if " OUT " in [message]
                   {
                      dissect {
                        mapping => {
                         message => "%{+Timestamp} %{+Timestamp} %{}Bytes %{Direction:} %{}SIP/2.0  %{Sip_Response} %{}<sip:%
{SipFromUser:}@%{}<sip:%{SipToUser:}@%{}CSeq:%{} %{Response_For_Msg:}:%{}"
                         }
                      }
                   }
                   else
                   { drop{} }
               }
               else if " IN " in [message]
               {
                  dissect {
                       mapping => {
                        message => "%{+Timestamp} %{+Timestamp} %{}Bytes %{Direction:} %{}<sip:%{SipToUser:}@%{}<sip:%{SipFromUser:}@%{}CSeq:%{} %{Response_For_Msg:}:%{}via:%{} %{ServerName:};%{}"
                     }
                  }
               }
               else if " OUT " in [message]
               {
                  dissect {
                      mapping => {
                          message => "%{+Timestamp} %{+Timestamp} %{}Bytes %{Direction:} %{}<sip:%{SipFromUser:}@%{}<sip:%{SipToUser:}@%{}CSeq:%{} %{Response_For_Msg:}:%{}"
                         }
                      }
               }

               else{
                  drop{}
               }
            }
         }
6

What does CPU usage look like on the Logstash host while processing?

7

Hi Chris

I tried in a diff VM with the below conf.

logstash version :-logstash 5.6.4

CentOS : Linux elkc-2 3.10.0-514.26.1.el7.x86_64 #1 SMP Thu Jun 29 16:05:25 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

H/W – 8 Core CPU , 32 Gb RAM

I placed 991Mb of XS files and processed them using Logstash, it took around 15 mins to process them and the data to be available in ES & Kibana

CPU utilization % - 639.5 MEM - 1.4

thanks
Arun

8

any help/comment on this?

9

It seems to be using a good amount of CPU without fully saturating it. What does the output config look like? What is the specification of the Elasticsearch cluster you are sending data to? What indexing rate are you seeing?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.