During setting up our ELK stack for application logs, some logs come structured and some aren't, I am running the JSON filter on every message to tag logs if they're structured or not.
After some errors happened and we weren't alerted by, I discovered that some logs were dropped without a warning or an error log (not even tagged with failure).
These messages are a valid JSON objects, but they include an array of empty objects.
I tried using the JSON codec for input but didn't work either and had the same behavior.
- Version: 7.3.0
- Operating System: Official Docker Image docker.elastic.co/logstash/logstash)
- Config File (if you have sensitive info, please remove it):
filter {
if [source] =~ /.*\.log$/ {
# Try to Parse as JSON, add tag unstrucutred on failure
json {
source => "message"
add_tag => "structured"
tag_on_failure => "unstructured"
skip_on_invalid_json => false
}
## .. .. .. rest of filters...
}
}
- Sample Data:
Oneliner (Dropped)
{"message":"Error Occured","context":[{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[{},"core.contact.created",{}]},{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[[[{},"onCoreContactCreated"]],"core.contact.created",{}]}],"level":400,"level_name":"ERROR","channel":"app","datetime":{"date":"2019-08-05 12:15:11.647910","timezone_type":3,"timezone":"UTC"}}
Readable (Dropped) (notice the empty objects)
{
"message": "Error Occured",
"context": [
{
"file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
"args": [
{},
"core.contact.created",
{}
]
},
{
"file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
"args": [
[
[
{},
"onCoreContactCreated"
]
],
"core.contact.created",
{}
]
}
],
"level": 400,
"level_name": "ERROR",
"channel": "app",
"datetime": {
"date": "2019-08-05 12:15:11.647910",
"timezone_type": 3,
"timezone": "UTC"
}
}
Oneliner (not dropped) (I strapped the empty objects)
{"message":"Error Occured","context":[{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":["core.contact.created"]},{"file":"/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php","args":[[["onCoreContactCreated"]],"core.contact.created"]}],"level":400,"level_name":"ERROR","channel":"app","datetime":{"date":"2019-08-05 12:15:11.647910","timezone_type":3,"timezone":"UTC"}}
Readable (not dropped)
{
"message": "Error Occured",
"context": [
{
"file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
"args": [
"core.contact.created"
]
},
{
"file": "/xxx/xxx/xxx.xxxx.xxxx/vendor/symfony/symfony/src/Symfony/Component/EventDispatcher/EventDispatcher.php",
"args": [
[
[
"onCoreContactCreated"
]
],
"core.contact.created"
]
}
],
"level": 400,
"level_name": "ERROR",
"channel": "app",
"datetime": {
"date": "2019-08-05 12:15:11.647910",
"timezone_type": 3,
"timezone": "UTC"
}
}
- Steps to Reproduce:
- Input JSON Message to Logstash with empty objects in an array
- No data sent to output and no warning/error messages outputted