Hi,
I'm running Filebeat and Logstash 5.6.5 and was having a bit of an issue with the json filter plugin provided for use with Logstash. I am pulling in multiline JSON logs with Filebeat, and I have an input configured and listening for beats on port 5044.
If I simply use:
codec => json
in the input file, then all is well. The individual fields are automatically parsed and viewable in Kibana, and things are fine. However, this is not optimal, because I have many different Beats, which may not all be using this codec.
The issue comes when I try to use the json filter in a filter configuration, such as
json {
source => "message"
}
(this is of course after tagging the event during the input phase, and using a conditional to grab it)
The log will make it into ES, but I will receive a _jsonparsefailure
tag and the json will reside in the message field, unparsed.
I've tried correcting errors I've received, in regard to brackets, by trying to remove the beginning and ending bracket(s) ([]
) in the source log file, but that doesn't seem to help, either.
Example of log file below:
[
{
"username": "test@gmail.com",
"access_device": {
"flash_version": "uninstalled",
"java_version": "uninstalled",
"os_version": "10",
"browser_version": "63.0.3239.132",
"os": "Windows",
"browser": "Chrome"
},
"timestamp": 1515709002,
"new_enrollment": false,
"ip": "1.1.1.1",
"integration": "Pass1",
"host": "supersecurity.com",
"reason": "User approved",
"eventtype": "auth",
"location": {
"city": "Shytown",
"state": "Upstate",
"country": "US"
},
"factor": "Push",
"device": "Test",
"result": "SUCCESS"
},
{
"username": "test@gmail.com",
"access_device": {
"flash_version": "uninstalled",
"java_version": "uninstalled",
"os_version": "10",
"browser_version": "63.0.3239.132",
"os": "Windows",
"browser": "Chrome"
},
"timestamp": 1515709665,
"new_enrollment": false,
"ip": "1.1.1.1",
"integration": "Pass1",
"host": "supersecurity.com",
"reason": "User approved",
"eventtype": "auth",
"location": {
"city": "Shytown",
"state": "Upstate",
"country": "US"
},
"factor": "Push",
"device": "Test",
"result": "SUCCESS"
}
]
Filebeat config:
filebeat.prospectors:
- input_type: log
paths:
- /var/log/auth.json
multiline.pattern: '^{'
multiline.negate: true
multiline.match: after
Any assistance would be greatly appreciated.
Thanks,
Cappy