Data loss using Json filter, messages dropped without any errors or warnings

Elastic Stack Logstash
1

During setting up our ELK stack for application logs, some logs come structured and some aren't, I am running the JSON filter on every message to tag logs if they're structured or not.

After some errors happened and we weren't alerted by, I discovered that some logs were dropped without a warning or an error log (not even tagged with failure).

These messages are a valid JSON objects, but they include an array of empty objects.

I tried using the JSON codec for input but didn't work either and had the same behavior.

filter {
  if [source] =~ /.*\.log$/ {
    # Try to Parse as JSON, add tag unstrucutred on failure
    json {
        source => "message"
        add_tag => "structured"
        tag_on_failure => "unstructured"
        skip_on_invalid_json => false
    }
    ## .. .. .. rest of filters...
  }
 }
  • Sample Data:

Oneliner (Dropped)

{"message":"Error Occured","context":[{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[{},"core.contact.created",{}]},{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[[[{},"onCoreContactCreated"]],"core.contact.created",{}]}],"level":400,"level_name":"ERROR","channel":"app","datetime":{"date":"2019-08-05 12:15:11.647910","timezone_type":3,"timezone":"UTC"}}

Readable (Dropped) (notice the empty objects)

{
    "message": "Error Occured",
    "context": [
        {
            "file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
            "args": [
                {},
                "core.contact.created",
                {}
            ]
        },
        {
            "file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
            "args": [
                [
                    [
                        {},
                        "onCoreContactCreated"
                    ]
                ],
                "core.contact.created",
                {}
            ]
        }
    ],
    "level": 400,
    "level_name": "ERROR",
    "channel": "app",
    "datetime": {
        "date": "2019-08-05 12:15:11.647910",
        "timezone_type": 3,
        "timezone": "UTC"
    }
}

Oneliner (not dropped) (I strapped the empty objects)

{"message":"Error Occured","context":[{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":["core.contact.created"]},{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[[["onCoreContactCreated"]],"core.contact.created"]}],"level":400,"level_name":"ERROR","channel":"app","datetime":{"date":"2019-08-05 12:15:11.647910","timezone_type":3,"timezone":"UTC"}}

Readable (not dropped)

{
    "message": "Error Occured",
    "context": [
        {
            "file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
            "args": [
                "core.contact.created"
            ]
        },
        {
            "file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
            "args": [
                [
                    [
                        "onCoreContactCreated"
                    ]
                ],
                "core.contact.created"
            ]
        }
    ],
    "level": 400,
    "level_name": "ERROR",
    "channel": "app",
    "datetime": {
        "date": "2019-08-05 12:15:11.647910",
        "timezone_type": 3,
        "timezone": "UTC"
    }
}
  • Steps to Reproduce:
  1. Input JSON Message to Logstash with empty objects in an array
  2. No data sent to output and no warning/error messages outputted
2

With this configuration in 7.3.0

input { generator { count => 1 lines => [ '{"message":"Error Occured","context":[{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[{},"core.contact.created",{}]},{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[[[{},"onCoreContactCreated"]],"core.contact.created",{}]}],"level":400,"level_name":"ERROR","channel":"app","datetime":{"date":"2019-08-05 12:15:11.647910","timezone_type":3,"timezone":"UTC"}}' ] } }
filter { json { source => "message" } }
output { stdout { codec => rubydebug } }

I get

{
"level_name" => "ERROR",
  "datetime" => {
             "date" => "2019-08-05 12:15:11.647910",
         "timezone" => "UTC",
    "timezone_type" => 3
},
"@timestamp" => 2019-08-07T14:55:30.288Z,
   "context" => [
    [0] {
        "file" => "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
        "args" => [
            [0] {},
            [1] "core.contact.created",
            [2] {}
        ]
    },
    [1] {
        "file" => "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
        "args" => [
            [0] [
                [0] [
                    [0] {},
                    [1] "onCoreContactCreated"
                ]
            ],
            [1] "core.contact.created",
            [2] {}
        ]
    }
],
   "message" => "Error Occured",
     "level" => 400,
   "channel" => "app"
}

Are you sending this data to elasticsearch? Are there errors in the elasticsearch logs?

1 Like
3

Yes, I just figured out it passes the JSON filter using stdout too! but doesn't pass the Elasticsearch Output tho. I don't know if it is something with Logstash's output plugin or from Elasticsearch side.

But there are no logs in either any of them.

4

I suggest you update the title and move the thread to the elasticsearch forum.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.