Logstash is taking long time to load config and run?

Elastic Stack Logstash
1

Hi,
My Logstash is taking 15 minutes to start if I have increased the size of config file .Althougth i have increasedsize of jvm heap not getting why this is happening.

2

Hi @RameshNagargoje,

Use VisualVM tool to troubleshoot the logstash. I hope it may help you.

Regards
Nikhil Kapoor

3

Perhaps related to https://github.com/elastic/logstash/issues/5507 or https://github.com/elastic/logstash/issues/6117.

4

Thanks @nikhil.k @magnusbaeck,
Is there size limit for Logstash config file? Because my config file is 14k lines if i have removed some lines from 14k to 1400 it loads but take some time .

5

I don't think there's a hard limit, but the config file parser might be so inefficient that such large files become unusable.

6

thank you @magnusbaeck ,

I am not getting how to optimize config file ,if we have 1k + regex to match and every regex match add some different tag

Adding Snippet of config

input
{

}
filter
{
if[logType] == "syslog"
{
grok
{
match => { message => "org.bluez.Error"}
add_tag => ["regexMatched"]
}
if "regexMatched" in [tags]
{
mutate
{
add_field => {"tagName"=>"Bluetooth_org.bluez.Error"}
add_field => {"module" => "comms"}
add_tag => ["SUCCESS"]
remove_tag =>"regexMatched"
}
}

    grok
    {
        match => { message => "hci0 command .* timeout"}
        add_tag => ["regexMatched"]
    }
    if "regexMatched" in [tags]
    {
        mutate
        {
            add_field  => {"tagName"=>"Bluetooth_hci0_command_timeout"}
            add_field  => {"module" => "comms"}
            add_tag => ["SUCCESS"]
            remove_tag =>"regexMatched"
        }
    }
     
    grok
    {
        match => { message => "Error resetting SDIO communications"}
        add_tag => ["regexMatched"]
    }
    if "regexMatched" in [tags]
    {
        mutate
        {
            add_field  => {"tagName"=>"Error resetting SDIO communications"}
            add_field  => {"module" => "comms"}
            add_tag => ["SUCCESS"]
            remove_tag =>"regexMatched"
        }
    }
     
    grok
    {
        match => { message => "\(NvCapture\) Error"}
        add_tag => ["regexMatched"]
    }
    if "regexMatched" in [tags]
    {
        mutate
        {
            add_field  => {"tagName"=>"NvCapture"}
            add_field  => {"module" => "camera"}
            add_tag => ["SUCCESS"]
            remove_tag =>"regexMatched"
        }
    }
     
    grok
    {
        match => { message => "\(Argus\) Error"}
        add_tag => ["regexMatched"]
    }
    if "regexMatched" in [tags]
    {
        mutate
        {
            add_field  => {"tagName"=>"Argus"}
            add_field  => {"module" => "camera"}
            add_tag => ["SUCCESS"]
            remove_tag =>"regexMatched"
        }
    }
     }

}
output{
}

7

grok
{
match => { message => "hci0 command .* timeout"}
add_tag => ["regexMatched"]
}
if "regexMatched" in [tags]
{
mutate
{
add_field => {"tagName"=>"Bluetooth_hci0_command_timeout"}
add_field => {"module" => "comms"}
add_tag => ["SUCCESS"]
remove_tag =>"regexMatched"
}
}

Shorter:

if [message] =~ /hci0 command .* timeout/ {
  mutate {
    add_field  => {
      "tagName"=>"Bluetooth_hci0_command_timeout"
      "module" => "comms"
    }
    add_tag => ["SUCCESS"]
  }
}

You should also be able to use a translate filter to list multiple regexps.

8

thank you @magnusbaeck it worked.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.