JSON Logs Parsed with JSON codec but not JSON filter

[cappy]

Hi,

I'm running Filebeat and Logstash 5.6.5 and was having a bit of an issue with the json filter plugin provided for use with Logstash. I am pulling in multiline JSON logs with Filebeat, and I have an input configured and listening for beats on port 5044.

If I simply use:

codec => json

in the input file, then all is well. The individual fields are automatically parsed and viewable in Kibana, and things are fine. However, this is not optimal, because I have many different Beats, which may not all be using this codec.

The issue comes when I try to use the json filter in a filter configuration, such as

json {
  source => "message"
} 

(this is of course after tagging the event during the input phase, and using a conditional to grab it)

The log will make it into ES, but I will receive a _jsonparsefailure tag and the json will reside in the message field, unparsed.

I've tried correcting errors I've received, in regard to brackets, by trying to remove the beginning and ending bracket(s) ([]) in the source log file, but that doesn't seem to help, either.

Example of log file below:

[
    {
        "username": "test@gmail.com", 
        "access_device": {
            "flash_version": "uninstalled", 
            "java_version": "uninstalled", 
            "os_version": "10", 
            "browser_version": "63.0.3239.132", 
            "os": "Windows", 
            "browser": "Chrome"
        }, 
        "timestamp": 1515709002, 
        "new_enrollment": false, 
        "ip": "1.1.1.1", 
        "integration": "Pass1", 
        "host": "supersecurity.com", 
        "reason": "User approved", 
        "eventtype": "auth", 
        "location": {
            "city": "Shytown", 
            "state": "Upstate", 
            "country": "US"
        }, 
        "factor": "Push", 
        "device": "Test", 
        "result": "SUCCESS"
    }, 
    {
        "username": "test@gmail.com", 
        "access_device": {
            "flash_version": "uninstalled", 
            "java_version": "uninstalled", 
            "os_version": "10", 
            "browser_version": "63.0.3239.132", 
            "os": "Windows", 
            "browser": "Chrome"
        }, 
        "timestamp": 1515709665, 
        "new_enrollment": false, 
        "ip": "1.1.1.1", 
        "integration": "Pass1", 
        "host": "supersecurity.com", 
        "reason": "User approved", 
        "eventtype": "auth", 
        "location": {
            "city": "Shytown", 
            "state": "Upstate", 
            "country": "US"
        }, 
        "factor": "Push", 
        "device": "Test", 
        "result": "SUCCESS"
    }
]

Filebeat config:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/auth.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

Any assistance would be greatly appreciated.

Thanks,
Cappy

[magnusbaeck (Magnus Bäck)]

The log will make it into ES, but I will receive a _jsonparsefailure tag and the json will reside in the message field, unparsed.

What's the contents of the message field? What error message does Logstash give (in its log)?

[cappy]

Thanks, Magnus. I ended up removing the commas, and inserting a newline after each entry in the log, then they were correctly parsed.

Ex.
{
"username": "test@gmail.com",
"access_device": {
"flash_version": "uninstalled",
"java_version": "uninstalled",
"os_version": "10",
"browser_version": "63.0.3239.132",
"os": "Windows",
"browser": "Chrome"
},
"timestamp": 1515709002,
"new_enrollment": false,
"ip": "1.1.1.1",
"integration": "Pass1",
"host": "supersecurity.com",
"reason": "User approved",
"eventtype": "auth",
"location": {
"city": "Shytown",
"state": "Upstate",
"country": "US"
},
"factor": "Push",
"device": "Test",
"result": "SUCCESS"
}
{
"username": "test@gmail.com",
"access_device": {
"flash_version": "uninstalled",
"java_version": "uninstalled",
"os_version": "10",
"browser_version": "63.0.3239.132",
"os": "Windows",
"browser": "Chrome"
},
"timestamp": 1515709002,
"new_enrollment": false,
"ip": "1.1.1.1",
"integration": "Pass1",
"host": "supersecurity.com",
"reason": "User approved",
"eventtype": "auth",
"location": {
"city": "Shytown",
"state": "Upstate",
"country": "US"
},
"factor": "Push",
"device": "Test",
"result": "SUCCESS"
}