Json forwarding (multiple fields)

[JensVanDeynse]

Hello

I'm trying to set up json forwarding from our filebeats to our logstash which will eventually forward it to the correct index in ElasticSearch. For several logs this works but one specific one fails for us and it looks like this:

{"bytes":0,"@timestamp":"2019-07-16T09:00:09.6338049+02:00","httpVersion":"HTTP/1.1","requestUrl":"/api/actuator/health","responseTime":1,"sourceIp":"10.7.0.4","status":200,"tenant":null,"user":null,"userAgent":"HAProxy_health_check","verb":"GET","correlationId":"41208d77","extraProperties":{"BuildNumber":"1.0.0.0","SourceContext":"Test.Net.Common.Logging.Helpers.Middlewares.AccessLoggingMiddleware"}}

I've tested several scenarios and concluded that it fails due to the 'extraProperties' field, when I copy/paste a line and edit it in the same file it does get send to the correct index. But the above log line is a valid JSON so I don't get why it fails and it also doesn't show up as failed since we check for the '_jsonparsefailure' tag in our logstash configuration.

The filebeat configuration looks as follows:

• type: log
  paths:
  • /var/log/test-session/access*.log
    fields:
    env: qa
    server: cordocker01.qa.test.be
    app: tarification-session-access
    logstash_template: access-logs
    fields_under_root: true
    json.keys_under_root: true
    json.add_error_key: true
    json.overwrite_keys: true

Has anybody else experienced this issue or is this just not possible?

Thank you for your help!