# Data loss using Json filter, messages dropped without any errors or warnings **URL:** \ **Category:** Logstash\ **Created:** [August 7, 2019, 2:13pm UTC](https://discuss.elastic.co/t/data-loss-using-json-filter-messages-dropped-without-any-errors-or-warnings/194255 "2019-08-07T14:13:33Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![sherifabdlnaby](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sherifabdlnaby/32/52395_2.png) [@sherifabdlnaby](https://discuss.elastic.co/u/sherifabdlnaby)\ **Post date:** [August 7, 2019, 2:13pm UTC](https://discuss.elastic.co/t/data-loss-using-json-filter-messages-dropped-without-any-errors-or-warnings/194255/1 "2019-08-07T14:13:33Z")
During setting up our ELK stack for application logs, some logs come structured and some aren't, I am running the JSON filter on every message to tag logs if they're structured or not. After some errors happened and we weren't alerted by, I discovered that some logs **were dropped** without a warning or an error log (not even tagged with failure). These messages are _a valid JSON_ objects, but they include an array of empty objects. I tried using the JSON codec for input but didn't work either and had the same behavior. - Version: 7.3.0 - Operating System: Official Docker Image [docker.elastic.co/logstash/logstash](http://docker.elastic.co/logstash/logstash)) - Config File (if you have sensitive info, please remove it): ```auto filter { if [source] =~ /.*\.log$/ { # Try to Parse as JSON, add tag unstrucutred on failure json { source => "message" add_tag => "structured" tag_on_failure => "unstructured" skip_on_invalid_json => false } ## .. .. .. rest of filters... } } ``` - Sample Data: #### Oneliner (Dropped) ```auto {"message":"Error Occured","context":[{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[{},"core.contact.created",{}]},{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[[[{},"onCoreContactCreated"]],"core.contact.created",{}]}],"level":400,"level_name":"ERROR","channel":"app","datetime":{"date":"2019-08-05 12:15:11.647910","timezone_type":3,"timezone":"UTC"}} ``` #### Readable (Dropped) (notice the empty objects) ```auto { "message": "Error Occured", "context": [ { "file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php", "args": [ {}, "core.contact.created", {} ] }, { "file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php", "args": [ [ [ {}, "onCoreContactCreated" ] ], "core.contact.created", {} ] } ], "level": 400, "level_name": "ERROR", "channel": "app", "datetime": { "date": "2019-08-05 12:15:11.647910", "timezone_type": 3, "timezone": "UTC" } } ``` * * * #### Oneliner (not dropped) (I strapped the empty objects) ```auto {"message":"Error Occured","context":[{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":["core.contact.created"]},{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[[["onCoreContactCreated"]],"core.contact.created"]}],"level":400,"level_name":"ERROR","channel":"app","datetime":{"date":"2019-08-05 12:15:11.647910","timezone_type":3,"timezone":"UTC"}} ``` #### Readable (not dropped) ```auto { "message": "Error Occured", "context": [ { "file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php", "args": [ "core.contact.created" ] }, { "file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php", "args": [ [ [ "onCoreContactCreated" ] ], "core.contact.created" ] } ], "level": 400, "level_name": "ERROR", "channel": "app", "datetime": { "date": "2019-08-05 12:15:11.647910", "timezone_type": 3, "timezone": "UTC" } } ``` - Steps to Reproduce: 1. Input JSON Message to Logstash with empty objects in an array 2. No data sent to output and no warning/error messages outputted ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 7, 2019, 2:58pm UTC](https://discuss.elastic.co/t/data-loss-using-json-filter-messages-dropped-without-any-errors-or-warnings/194255/2 "2019-08-07T14:58:05Z")
With this configuration in 7.3.0 ``` input { generator { count => 1 lines => ['{"message":"Error Occured","context":[{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[{},"core.contact.created",{}]},{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[[[{},"onCoreContactCreated"]],"core.contact.created",{}]}],"level":400,"level_name":"ERROR","channel":"app","datetime":{"date":"2019-08-05 12:15:11.647910","timezone_type":3,"timezone":"UTC"}}' ] } } filter { json { source => "message" } } output { stdout { codec => rubydebug } } ``` I get ``` { "level_name" => "ERROR", "datetime" => { "date" => "2019-08-05 12:15:11.647910", "timezone" => "UTC", "timezone_type" => 3 }, "@timestamp" => 2019-08-07T14:55:30.288Z, "context" => [ [0] { "file" => "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php", "args" => [ [0] {}, [1] "core.contact.created", [2] {} ] }, [1] { "file" => "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php", "args" => [ [0] [ [0] [ [0] {}, [1] "onCoreContactCreated" ] ], [1] "core.contact.created", [2] {} ] } ], "message" => "Error Occured", "level" => 400, "channel" => "app" } ``` Are you sending this data to elasticsearch? Are there errors in the elasticsearch logs? ---
**Author:** ![sherifabdlnaby](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sherifabdlnaby/32/52395_2.png) [@sherifabdlnaby](https://discuss.elastic.co/u/sherifabdlnaby)\ **Post date:** [August 7, 2019, 3:31pm UTC](https://discuss.elastic.co/t/data-loss-using-json-filter-messages-dropped-without-any-errors-or-warnings/194255/3 "2019-08-07T15:31:26Z")
Yes, I just figured out it passes the JSON filter using `stdout` too! but doesn't pass the Elasticsearch Output tho. I don't know if it is something with Logstash's output plugin or from Elasticsearch side. But there are no logs in either any of them. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 7, 2019, 3:48pm UTC](https://discuss.elastic.co/t/data-loss-using-json-filter-messages-dropped-without-any-errors-or-warnings/194255/4 "2019-08-07T15:48:50Z")
I suggest you update the title and move the thread to the elasticsearch forum. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 4, 2019, 3:57pm UTC](https://discuss.elastic.co/t/data-loss-using-json-filter-messages-dropped-without-any-errors-or-warnings/194255/5 "2019-09-04T15:57:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.