Timeout executing grok & Value too large to output

Elastic Stack Logstash
1

Hello,

I've already read a lot about that problem but I really don't know what to do precisely.

I'm using ELK 5.4.1 since a week now and after adding some grok filters, there is a tons of Timeout on my logstash-plain.log.

[2017-08-09T10:28:00,898][WARN ][logstash.filters.grok ] Timeout executing grok '%{DATA:date}\sDEBUG%{DATA:body}\s-\scategory=%{GREEDYDATA:category}' against field 'message' with value 'Value too large to output (25308 bytes)! First 255 chars are: 2017-08-09 10:19:43,388 DEBUG [http-nio-8080-exec-1533] [com.services.aggregator.ServicesAggregationStrategy] - oldJson={"3":{"id":"3","name":"small data","layer_info_url":"an url'!

The grok filter is :

message","%{DATA:date}\sDEBUG%{DATA:body}\s-\scategory=%{GREEDYDATA:category}",

The problem is my CPU usage wich is at 99%.

Could you helped me, to understand what's the problem ? My grok filter ? The configuration (have been done with bitnami) ?

Thanks by advance,
Slop

2

DATA and GREEDYDATA patterns are very expensive. Switch to something more specific (perhaps NOTSPACE?). It's extremely rare to have to use more than one DATA or GREEDYDATA in the same grok expression.

3

Thanks for that answer @magnusbaeck, I've just updated grok filter done by someone else to fetch "category".

Everything else on the grok filter is trash for me, what should I do to fetch "category" attribut and ignore everything else ?

4

What does a log entry look like? What fields (and values) do you want to extract?

5

The log look like :

2017-08-09 11:53:14,895 DEBUG [http-nio-8080-exec-1571] [com.services.processor.SolrDistributionsRoutePreprocessor] - category=report

And I just want to extract category=report

6

In that case I suppose \scategory=%{GREEDYDATA:category} should work.

7

Ok, it work, but in some case, there's data after category=report. And with ur solution, I fetch all those data, but I only want category=report.

Is there a solution for that ? To just fetch category=report and ignore everything else ?

8

You can use NOTSPACE instead of GREEDYDATA, assuming the category string won't ever contain a space.

9

It didn't work, here's an exemple :

2017-08-09 09:06:31,884 INFO [http-nio-8080-exec-1357] [com.client.AbstractClient] - Client url is http://localhost:8080/services/search/distributions?mtd_id=658874115&category=block&object_type_list=["basin"%2C"company"%2C"country"]&fq=distributions.id%3Ablock-100000237163&_=1502258193028

with this filter : category=%{NOTSPACE:category}

The result is :

{
"category": [
[
"block&object_type_list=%5B%22basin%22%2C%22company%22%2C%22country%22%5D&fq=distributions.id%3Ablock-100000237163&_=1502258193028"
]
]
}

10

I don't have time to help out with new suggestions if you keep on changing the text you want to match.

11

I'm sorry but I havn't post all the different case I had... because they are many.

Something I don't understand is, with grok debugger (https://grokdebug.herokuapp.com/) the results look ok.
But the render on kibana isn't .

Here's a full exemple :

my log

2017-08-10 11:46:00,185 INFO [http-nio-8080-exec-1998] [com.api.client.AbstractClient] - Client url is http://localhost:8080/services/search/distributions?mtd_id=-report-4774489&category=report&object_type_list=["basin"%2C"block"%2C"country"%2C"field"%2C"well"]&fq=distributions.id%3Aesearch-report-696807&_=1502353506766

my filter

&category=%{NOTSPACE:category}&object_type_list=%{GREEDYDATA:trash}

The grok debugger result

{
"category": [
[
"report"
]
],
"trash": [
[
"%5B%22basin%22%2C%22block%22%2C%22country%22%2C%22field%22%2C%22well%22%5D&fq=distributions.id%3Aeureka-esearch-report-696807&_=1502353506766"
]
]
}

And the kibana result

report&object_type_list=%5B%22basin%22%2C%22block%22%2C%22country%22%2C%22field%22%2C%22well%22%5D&fq=distributions.id%3Aeureka-esearch-report-696807&_=1502353506766

I really don't understand ....

But thanks for your help

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.