Json filter dropping messages

[ikirani (pan)]

I have a set of 100 json messages that i send through Logstash Json filter.

Each and every time I see a variable number of messages persisted to output file, anywhere between 60 to 100.

Things I've tried so far:

1. Upgrade to 5.5 and enable dead_letter_queue (no messages are written to this queue, folder is empty). I don't see anything here even when I send an invalid json message!
2. Applied a tag on failure called "failed" and tried to write those messages to a new file. That file remains empty too.
3. look for messages with tag _jsonparsefailure. No messages show up with that either.

All the 100 messages have a valid JSON structure.

Here is my configuration:

input{
file{ path => /path/to/input/file/with/line-delimited/jsons }
}

filter{
json{
source => message
}
}

output{
file { path => /path/to/output/file}
}

On 10 consecutive runs of the exact same file of 100 messages, the output file had following number of messages:
69, 93, 97, 99, 100, 87, 64, 100, 92, 91

Versions of Logstash I've tried so far: 5.4.1 and 5.5.2

The only complexity to a regular use-case is one of the fields in json is a json object, escaped. I can provide a sample via email, cannot post here.

Please help me debug this as adding tags, writing to DLQ all failed.

[rodelor97 (Robert de)]

I am seeing the same issue on my local instance. Is this a known bug? We use the json filter in at least half of our messages, so this is a blocker.

[ikirani (pan)]

A simpler set of data (200 messages) with no other nested complications.... that I repeatedly run through the json filter.

At the end of 20 consequtive runs, I expect 4000 messages in output.log locally where the output filter writes to. But I consistently see a lower count.

No error messages in the Logstash log, nothing showing up in --debug either. But the behavior is consistent.

sample: {"place":"someplace","which":"someApp","host":"someHost","someTime":1504914709800,"ms":"1504914703611","path":"/data/logs/someApp/someApp.log","log.lvl":"WARN","someplace":"P1","type":"FL","exc":"No Exception","msg":"08 Sep 2017 19:51:43,611 WARN [2132093549@qtp-2130427677-7505] (com.somewhere.someApp.source.HTTPSource$someAppHTTPServlet.doPost:276) - requestHeaders: {Accept-Language=en-US,en;q=0.8, Host=myApp.somewhere.net, Content-Length=1136, Accept-Encoding=gzip, deflate, br, X-Forwarded-For=67.10.58.128, User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) whichleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36, Origin=https://auth.somewhere.com, Accept=/, Connection=Keep-Alive, Content-Type=whichlication/csp-report, X-BlueCoat-Via=8d2aff7ec5959423}"}

[ikirani (pan)]

Should this be a Bug? Where can I report bugs for Logstash filters? This is a blocker for us moving into Production.

[ikirani (pan)]

Hi,

Is there an older version of Logstash that we could use without the above reported issue? Any work-around or triage on this problem will really help. We are blocked at QA and need to ensure 100% throughput to meet our SLA.