Logtash ignores some lines randomly

Logtash apparently ignores some of my log lines done by nginx.

I'm trying to parse my customized nginx logs with logstash, I'm adding the response time by upstream, etc... and a few more fields that aren't normally on the nginx logs but have a lot of interest.

But somehow logstash seems to ignore some lines, even the grok debugger (https://grokdebug.herokuapp.com/) doesn't seem do discard them.

Following is the ignored line that apparently match with the first posibility:

"08/Jan/2020:14:22:47 +0000" client=88.217.181.177 method=POST request="POST /common/message_count HTTP/2.0" request_length=402 status=200 bytes_sent=884 body_bytes_sent=90 referer=https://backoffice.andronautic.com/planner user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0" upstream_addr=unix:/var/run/popeye/back.socket upstream_status=200 request_time=0.731 ssl_session_reused=r upstream_response_time=0.732 upstream_connect_time=0.000 upstream_header_time=0.732

And next the logstash.conf that I use, attached on pastebin due to weird pasting here:

https://pastebin.com/L5vQzbMU

Why is it ignored on logstash/kibana? /tmp/grok_failures is empty for hours... but still, ignoring some lines :frowning:

Many thanks!

I cannot say why some lines are ignored but you can simplify your configuration by having grok do the conversions using, for example,

bytes_sent=(?:%{NUMBER:[nginx][access][bytes_sent]:int}|-)
upstream_header_time=(?:%{NUMBER:[upstream][access][header_time]:float}|-)

Personally I wouldn't bother converting something like status_code to an integer, since it is not helpful to be able to calculate the average status code, or the total of status codes over a set of requests.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.