Timeout executing grok

Elastic Stack Logstash
1

Hello I am getting a timeout issue when running a grok filter. My logstash config looks like so.

filter {
if [type] == "cast" {
grok {
break_on_match => false

             match => { 'message'=> '\[%{DATA:timestamp}\]  \[%{DATA:pool}\] %{WORD:ignore} %{NUMBER:pid}'}

             match => { 'message' => '%{DATESTAMP:timestamp} \[%{WORD:status}\] %{DATA:ignore}\: \*%{NUMBER:ignore} %{GREEDYDATA:error-message}'}

             match => { 'message' => '%{IPORHOST:ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:ignore} %{PATH:a-info}%{DATA:ignore}&%{WORD:ignore}=%{NUMBER:font}%{DATA:ignore}" %{INT:http_response} %{INT:wall} "-" %{DATA:ignore}\(%{WORD:OS}; %{DATA:ignore}\) %{WORD:browser}%{GREEDYDATA:ignore}'}

             match => { 'message' => '%{IPORHOST:ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:ignore} %{PATH:a-info}%{DATA:ignore}&%{WORD:ignore}=%{NUMBER:font}%{DATA:ignore}" %{INT:http_response} %{INT:wall} "%{URI:URL}" %{DATA:ignore}\(%{WORD:OS}%{DATA:ignore},%{DATA:ignore}\) %{WORD:browser}%{GREEDYDATA:ignore}'}

}

I checked the message it is time outing on and it matches on of the filters above. I dont get why it would timeout. Any help would be greatly appreciated.

2

just updated stack to latest and greatest, still the same issue.

3

ive tried adding tag_n_timeout and then grab that time out and jsut grok match it to %{GREEDYDATA:timeout} and it doesn't work.

4

What is the exact error, please post the log.

5

Timeout executing grok '%{IPORHOST:ip} - - [%{HTTPDATE:timestamp}] "%{WORD:ignore} %{PATH:a-info}%{DATA:ignore}&%{WORD:ignore}=%{NUMBER:font}%{DATA:ignore}" %{INT:http_response} %{INT:wall} "-" %{DATA:ignore}(%{WORD:OS}; %{DATA:ignore}) %{WORD:browser}%{GREEDYDATA:ignore}' against field 'message' with value 'Value too large to output (431 bytes)! First 255 chars are:

6

anyone have an idea of what can be causing this?

7

added x-pack to logstash and see that CPU is at 95-97% is this because of the bad filter or could this be causing it?

8

Does monitoring show you where it's spending most of its time?

9

it does not, unless i am looking in the wrong place?

10

You have a few occurrences of DATA and GREEDYDATA that tend to match a lot and can be expensive/slow to compute. Try to be more specific. It also looks like you have a few fields named ignore. If you do not want to capture the data, I believe you can simply just not name it.

11

using greedydata is just to grab the rest of the log? Is there a better work around than grabbing each piece of data?

12

The last GREEDYDATA is fine, but you may want to be more specific and replace the DATA patterns earlier in the pattern. If you show what a log line looks like we may be able to provide better guidance.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.