Decode_json_fields not processing nested array?

[adamparmelee]

Hi. I've been struggling with this for a bit now. Nothing I've tried from forum scouring has done the trick.

here's a log entry:

{"transaction":{"client_ip":"x.x.x.x","time_stamp":"Wed Jul 15 19:08:29 2020","server_id":"6fe897d55b2c6b48f3beb2047585a915d00a5d97","client_port":0,"host_ip":"10.10.1.39","host_port":8080,"unique_id":"1594840109","request":{"method":"GET","http_version":1.1,"uri":"/","headers":{"host":"0","Accept":"*/*","User-Agent":"curl/7.64.1","X-Forwarded-For":"x.x.x.x","X-Forwarded-Port":"443","X-Forwarded-Proto":"https","Connection":"keep-alive"}},"response":{"body":"","http_code":418,"headers":{"Server":"nginx","Date":"Wed, 15 Jul 2020 19:08:29 GMT","Connection":"keep-alive"}},"producer":{"modsecurity":"ModSecurity v3.0.4 (Linux)","connector":"ModSecurity-nginx v1.0.1","secrules_engine":"Enabled","components":["OWASP_CRS/3.0.2\""]},"messages":[{"message":"Host header is a numeric IP address","details":{"match":"Matched \"Operator `Rx' with parameter `^[\\d.:]+$' against variable `REQUEST_HEADERS:host' (Value: `0' )","reference":"o0,1v21,1","ruleId":"920350","file":"/etc/nginx/modsec/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf","lineNumber":"777","data":"0","severity":"4","ver":"OWASP_CRS/3.0.0","rev":"2","tags":["application-multi","language-multi","platform-multi","attack-protocol","OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST","WASCTC/WASC-21","OWASP_TOP_10/A7","PCI/6.5.10"],"maturity":"9","accuracy":"9"}}]}}

and this is my config:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/modsec_audit.log
  tags: ["modsecurity"]
  processors:
    - decode_json_fields:
        fields: ["message"]
        process_array: true
        max_depth: 6
        target: ""
        overwrite_keys: false

I'm getting partial decode, but the transaction.messages entry is left as a json object.

could it be related to this?

[ChrsMark (Chris Mark)]

Hi!

It could be related to the issue you mentioned! Do you have a complete input/output example to share? If so feel free to post your case in https://github.com/elastic/beats/issues/19830 so as to provide more content to the team and help fixing it!

C.

[adamparmelee]

thanks! by output you mean from the debug logs?