Grok gives an error, causing cpu to be high

Elastic Stack Logstash
1

Hello, I don't know why I set the matching specifications, logstash keeps getting error log and cpu usage rate rises rapidly.

The error log is as follows:
[WARN ][logstash.filters.grok ] Timeout executing grok '(.?)"%{GREEDYDATA:timestamp}" fw=%{HOSTNAME:HostName}(.?)user=%{USERNAME:UserName}(.?)%{IP:SourceAddress} op="%{USERNAME:UserLogging}"(.?)user login(.*?)' against field 'message' with value 'Value too large to output (534 bytes)! First 255 chars are:

A single sample log message is as follows:
id=tos time="2008-5-14 06:53:10" fw=TopsecOS pri=6 type=ips recorder=IPSAR proto=tcp src=192.168.3.2 sport=80 dst=192.168.2.2 dport=69000 rule= repeat= msg= appendix= application="qq" op="block" interface= sdev=eth10 ddev=eth11
id=tos time="2008-5-14 06:53:10" fw=TopsecOS pri=6 type=ips recorder=IPSAR proto=tcp src=192.168.3.2 sport=80 dst=192.168.2.2 dport=69000 rule= repeat= msg= appendix= application="qq" op="block" interface= sdev=eth10 ddev=eth11
id=tos time="2008-5-14 06:53:10" fw=TopsecOS pri=6 type=ips recorder=IPSAR proto=tcp src=192.168.3.2 sport=80 dst=192.168.2.2 dport=69000 rule= repeat= msg= appendix= application="qq" op="block" interface= sdev=eth10 ddev=eth11
id=tos time="2008-5-14 06:53:10" fw=TopsecOS pri=6 type=ips recorder=IPSAR proto=tcp src=192.168.3.2 sport=80 dst=192.168.2.2 dport=69000 rule= repeat= msg= appendix= application="qq" op="block" interface= sdev=eth10 ddev=eth11

2

are you sure your using correct grok pattern and let me know what value you need to extract

3

There is also no prompt problem with the grok online matching test. What do you mean by the value?

4

which message your try to extract with this pattern

5

Still keep reporting errors, can you help me write a change and let me test it?

6

can you share your logstash config file

7

Patterns that start with GREEDYDATA are very expensive because they have to backtrack and retry so much. This can result in timeouts.

Why not use

kv { whitespace => "strict" }

which will parse the first line as

   "sequence" => 0,
        "pri" => "6",
        "dst" => "192.168.2.2",
      "proto" => "tcp",
      "sport" => "80",
      "dport" => "69000",
"application" => "qq",
         "op" => "block",
       "ddev" => "eth11",
       "time" => "2008-5-14 06:53:10",
       "type" => "ips",
         "fw" => "TopsecOS",
   "recorder" => "IPSAR",
       "sdev" => "eth10",
         "id" => "tos",
        "src" => "192.168.3.2"

Empty fields such as rule, repeat, msg and appendix are discarded.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.