Value too large to output (365 bytes)

Elastic Stack Logstash
1

Good Afternoon,

I finally presented my final grade work with ELK, and i'm quite happy of the results. By the way, we're adding new features when it talks to data analysis and painting graphics.

One of the problems that i got stucked in is when logstash isn't able to parse an output larger than 255 bytes (maybe i'm wrong explaining it).

The thing, is that the output error says that Value is too large to output.

[WARN ] 2018-07-03 16:55:43.414 [Ruby-0-Thread-19@[main]>worker1: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385] grok - Timeout executing grok '%{IP:clientip} - "%{NOTSPACE:username}" [%{HTTPDATE:http_date}] "%{NOTSPACE:method} %{PATH:path} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NOTSPACE:bytes} "%{NOTSPACE:referrer}" %{QUOTEDSTRING:user_agent}' against field 'message' with value 'Value too large to output (352 bytes)! First 255 chars are: X.X.X.X - "user.name" [27/Jun/2016:23:56:03 +0200] "GET /forum/rss/recentTopics.page.key?KEY=xxxxxxxxxxxxxxxxxxx HTTP/1.1" 200 5661 "https://URLwithsensibledata" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKi'!

I've been looking for solutions into discuss.elastic.co, and many of them were because of the use of patterns like DATA or GREEDYDATA, or due to timeout wich i tried to change, and sometimes does work, and sometimes not.

My match => { "message" => looks like this:

match => { "message" => '%{IP:clientip} - "%{NOTSPACE:username}" [%{HTTPDATE:http_date}] "%{NOTSPACE:method} %{PATH:path} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NOTSPACE:bytes} "%{NOTSPACE:referrer}" %{QUOTEDSTRING:user_agent}' }

At the beginning, i was using DATA, and GREEDYDATA. and tried to reduce them until i got none of it, i've been looking which of those i use are also expensive, and i think HTTPDATE is it quite, but not sure if this would be my error.

My question woul be, this warning "Value too large to output" is because of log size, or because 'timestamp' ? Should i use filebeat to parse this data?

Thank you very much,

Toni.

2

The underlying error that you need to fix is the timeout in grok. One thing you can do to help with that is to anchor your pattern (i.e. start it with ^, which matches start of line).

It logs the field that it is matching against to help you diagnose the problem, but does not want to fill the log with extremely large fields, so it truncates it at 255 characters and is just letting you know that the log file entry is truncated.

In Kibana, you can search for tags: _groktimeout and see the entire message.

1 Like
4

Hello Badger,

That solves the problem when matching the start of the line, but the thing is that i would like to parse the line, and automatically correlate the data into variables so i can draw graphs.

Thanks for the answer by the way!

5

Can you give an example of a complete line that is not getting parsed and the grok pattern that does not match it?

6

Hey! i just solve it!

It was because of a missmatching of the URL, actually when matching the URL i thought it was getting it right (kind of) but it was only getting the Path as it was a system path, and not an URL/URI Path so when it finds an '?' stops.
I changed the patter PATH for URIPATHPARAM and now is able to parse data fine and no timeout is happening anymore.

Thank you by the way @Badger!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.