Logtash ignores some lines randomly

[JustDevZero (Daniel Ripoll)]

Logtash apparently ignores some of my log lines done by nginx.

I'm trying to parse my customized nginx logs with logstash, I'm adding the response time by upstream, etc... and a few more fields that aren't normally on the nginx logs but have a lot of interest.

But somehow logstash seems to ignore some lines, even the grok debugger (https://grokdebug.herokuapp.com/) doesn't seem do discard them.

Following is the ignored line that apparently match with the first posibility:

"08/Jan/2020:14:22:47 +0000" client=88.217.181.177 method=POST request="POST /common/message_count HTTP/2.0" request_length=402 status=200 bytes_sent=884 body_bytes_sent=90 referer=https://backoffice.andronautic.com/planner user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0" upstream_addr=unix:/var/run/popeye/back.socket upstream_status=200 request_time=0.731 ssl_session_reused=r upstream_response_time=0.732 upstream_connect_time=0.000 upstream_header_time=0.732

And next the logstash.conf that I use, attached on pastebin due to weird pasting here:

input { beats { port => 5044 }}filter { if "beats_input_code -...

Why is it ignored on logstash/kibana? /tmp/grok_failures is empty for hours... but still, ignoring some lines

Many thanks!

[Badger]

I cannot say why some lines are ignored but you can simplify your configuration by having grok do the conversions using, for example,

bytes_sent=(?:%{NUMBER:[nginx][access][bytes_sent]:int}|-)
upstream_header_time=(?:%{NUMBER:[upstream][access][header_time]:float}|-)

Personally I wouldn't bother converting something like status_code to an integer, since it is not helpful to be able to calculate the average status code, or the total of status codes over a set of requests.