How grok filter use more cpu?

Elastic Stack Logstash
1

Hi, I have used grok filter to match the first line after the second line, When config file is loaded then it used more CPU I am not getting why it is happening

here is my grok pattern

(?<Matched_Lines>hd_bus_rxctl: rxcnt_timeout(.*\n)+(?i).*dhd_check_hang: Event HANG send up due to)

2

I have used grok filter to match the first line after the second line

What do you mean? Please show your full configuration and example input.

3

here is input:

Expressions can be long and complex. Expressions can contain other expressions, you can negate expressions with !, and you can group them with parentheses (...).

For example, the following conditional uses the mutate filter to remove the field secret if the field action has a value of login:


rxctl: rxcnt_timeout=5, rxlen=0
15119.753889] [<c0064364>] (kthread+0xe0/0xe4) from [<c000f1a0>] (ret_from_fork+0x14/0x20)
15119.753889] dhd_bus_rxctl: rxcnt_timeout=5, rxlen=0
15119.753889] dhd_bus_rxctl: rxcnt_timeout=5, rxlen=0
15119.753889] dhd_bus_rxctl: rxcnt_timeout=5, rxlen=0
15119.753889] dhd_bus_rxctl: rxcnt_timeout=5, rxlen=0

[15119.753902] dhd_check_hang: Event HANG send up due to  re=5 te=0 e=-110 s=2
[15119.753902] Dhd_check_hang: Event HANG send up due to  re=5 te=0 e=-110 s=2
[15119.753902] Dhd_check_hang: Event HANG send up due to  re=5 te=0 e=-110 s=2
[15119.753902] Dhd_check_hang: Event HANG send up due to  re=5 te=0 e=-110 s=2
[15119.753917] Dhd_check_hang: Event HANG send up due to  re=5 te=0 e=-110 s=2

[15119.753937] Dhd_prot_ioctl : bus is down. we have nothing to do
[15119.791431] [<c0ab1378>] (schedule_timeout+0x158/0x25c) from [<ea1e0000>] (0xea1e0000)
[15119.799331] kworker/3:2     R running      0 29597      2 0x00000000
[15119.805699] [<c0ab2fd4>] (__schedule+0x3d0/0x8a4) from [<c005d71c>] (worker_thread+0x1fc/0x3dc)
[15119.814384] [<c005d71c>] (worker_thread+0x1fc/0x3dc) from [<c0064364>] (kthread+0xe0/0xe4)
[15119.822637] [<c0064364>] (kthread+0xe0/0xe4) from [<c000f1a0>] (ret_from_fork+0x14/0x20)
[15119.830710] kworker/u8:1    S c0ab2fd4     0 29738      2 0x00000000
[15119.837078] [<c0ab2fd4>] (__schedule+0x3d0/0x8a4) from [<c005d71c>] (worker_thread+0x1fc/0x3dc)
[15119.845763] [<c005d71c>] (worker_thread+0x1fc/0x3dc) from [<c0064364>] (kthread+0xe0/0xe4)
[15119.854015] [<c0064364>] (kthread+0xe0/0xe4) from [<c000f1a0>] (ret_from_fork+0x14/0x20)
[15119.862088] kworker/u8:4    S c0ab2fd4     0 29739      2 0x00000000
[15119.868455] [<c0ab2fd4>] (__schedule+0x3d0/0x8a4) from [<c005d71c>] (worker_thread+0x1fc/0x3dc)
[15119.877140] [<c005d71c>] (worker_thread+0x1fc/0x3dc) from [<c0064364>] (kthread+0xe0/0xe4)
[15119.885391] [<c0064364>] (kthread+0xe0/0xe4) from [<c000f1a0>] (ret_from_fork+0x14/0x20)
[15119.893468] Sched Debug Version: v0.10, 3.10.96+ #1
[15119.898337] ktime

here is config file

if[type] == 'dmesg'
{

  grok
  {
    match => { logs => "(?<Matched_Lines>.*hd_bus_rxctl: rxcnt_timeout.*(.*\n)+.*(?i)dhd_check_hang: Event HANG send up due to.*)"}
    add_tag => ["regexMatched"]      
  }

  if "regexMatched" in [tags]
  { 

          ruby
          {
            code => 'event.set("logSnippet", event.get("logs").scan(/.*(?i)dhd_check_hang: Event HANG send up due to.*/))'
          }
          mutate
          {
            add_field  => {"tagName"=>"GENERIC_EMERGENCY"}
            add_field  => {"module" => "null"}
            add_tag => ["SUCCESS"]
            remove_tag =>"regexMatched"
          }
       
  }

}

when config is loaded then cpu sounds starts increasing

here is error is displayed.

For example, the following conditional uses the mutate filter to remove the field se'!

[2018-05-02T02:17:26,666][WARN ][logstash.filters.grok ] Timeout executing grok '(?<Matched_Lines>.hd_bus_rxctl: rxcnt_timeout.(.\n)+.(?i)dhd_check_hang: Event HANG send up due to.*)' against field 'logs' with value 'Value too large to output (2470 bytes)! First 255 chars are:
Expressions can be long and complex. Expressions can contain other expressions, you can negate expressions with !, and you can group them with parentheses (...).

4

@magnusbaeck, please have a look

5

It looks like you want to capture all lines between the lines matching two regular expressions. Doing this with a regexp isn't very efficient. Doing it in a ruby filter that loops over the lines in the field should be much faster.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.