Drop beginning of JSON log

[SantZP]

Hi!
Im trying to parse the following JSON log in FileBeat but i keep getting error.message: "Error decoding JSON: invalid character 'J' looking for beginning of value: "

JSON Authentication: {"timestamp": "2019-06-04T10:15:23.174830-0300", "type": "Authentication", "Authentication": {"version": {"major": 1, "minor": 0}, "status": "NT_STATUS_OK", "localAddress": null, "remoteAddress": "ipv4:ip:port", "serviceDescription": "Kerberos KDC", "authDescription": "ENC-TS Pre-authentication", "clientDomain": null, "clientAccount": "account@domain", "workstation": null, "becameAccount": "account", "becameDomain": "DOMAIN", "becameSid": "S-1-5-21-310888293-2901519497-23123456789-1113", "mappedAccount": "account", "mappedDomain": "DOMAIN", "netlogonComputer": null, "netlogonTrustAccount": null, "netlogonNegotiateFlags": "0x00000000", "netlogonSecureChannelType": 0, "netlogonTrustAccountSid": null, "passwordType": "aes256-cts-hmac-sha1-96", "duration": 9595}}

I assume i need to drop "JSON Authentication: " so that it becomes a valid JSON.

Filebeat.yml:

    filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /opt/samba4/var/log/samba/*.samba
      json.message_key: log
      json.keys_under_root: true
      json.add_error_key: true

    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~
      - decode_json_fields:
         fields: ['message']
         target: json

Is there a solution? Thx!

[martinr_ubi (Martin R.)]

I think i can help but please share the version of filebeat you’re using.

[SantZP]

Sure!

# filebeat version
filebeat version 7.0.0 (amd64), libbeat 7.0.0 [da192b7d09af1d735cef19ea7816b8b8a5d4a323 built 2019-04-05 22:11:42 +0000 UTC]

[martinr_ubi (Martin R.)]

Hi,

My testing floor Filebeat config:

filebeat.inputs:
- type: log
  paths:
    - /tmp/inputs_files/*
  processors:
    - dissect:
        tokenizer: "JSON Authentication: %{json_string}"
        field: "message" # we're verbose for clarity but the default is "message"
        target_prefix: "" # put the json_string field at the root of the document
    - decode_json_fields:
        fields: ["json_string"]
        target: "json"
output.file:
  path: "/output/filebeat"
  filename: filebeat
  permissions: 0777
  codec.json:
    pretty: true
setup.template.enabled: false
xpack.monitoring.enabled: false
logging.json: true
logging.level: info

Content of /tmp/inputs_files/test.log:

JSON Authentication: {"timestamp": "2019-06-04T10:15:23.174830-0300", "type": "Authentication", "Authentication": {"version": {"major": 1, "minor": 0}, "status": "NT_STATUS_OK", "localAddress": null, "remoteAddress": "ipv4:ip:port", "serviceDescription": "Kerberos KDC", "authDescription": "ENC-TS Pre-authentication", "clientDomain": null, "clientAccount": "account@domain", "workstation": null, "becameAccount": "account", "becameDomain": "DOMAIN", "becameSid": "S-1-5-21-310888293-2901519497-23123456789-1113", "mappedAccount": "account", "mappedDomain": "DOMAIN", "netlogonComputer": null, "netlogonTrustAccount": null, "netlogonNegotiateFlags": "0x00000000", "netlogonSecureChannelType": 0, "netlogonTrustAccountSid": null, "passwordType": "aes256-cts-hmac-sha1-96", "duration": 9595}}

Resulting content in /output/filebeat/filebeat:

{
  "@timestamp": "2019-06-06T06:07:05.488Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.0.0"
  },
  "agent": {
    "id": "37ddd7d3-90a7-44a7-ab5d-577c148084ff",
    "version": "7.0.0",
    "type": "filebeat",
    "ephemeral_id": "61b3a4cc-86b6-44fe-b2dd-ce7462998006",
    "hostname": "44eb8f3d7558"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/tmp/inputs_files/test.log"
    }
  },
  "message": "JSON Authentication: {\"timestamp\": \"2019-06-04T10:15:23.174830-0300\", \"type\": \"Authentication\", \"Authentication\": {\"version\": {\"major\": 1, \"minor\": 0}, \"status\": \"NT_STATUS_OK\", \"localAddress\": null, \"remoteAddress\": \"ipv4:ip:port\", \"serviceDescription\": \"Kerberos KDC\", \"authDescription\": \"ENC-TS Pre-authentication\", \"clientDomain\": null, \"clientAccount\": \"account@domain\", \"workstation\": null, \"becameAccount\": \"account\", \"becameDomain\": \"DOMAIN\", \"becameSid\": \"S-1-5-21-310888293-2901519497-23123456789-1113\", \"mappedAccount\": \"account\", \"mappedDomain\": \"DOMAIN\", \"netlogonComputer\": null, \"netlogonTrustAccount\": null, \"netlogonNegotiateFlags\": \"0x00000000\", \"netlogonSecureChannelType\": 0, \"netlogonTrustAccountSid\": null, \"passwordType\": \"aes256-cts-hmac-sha1-96\", \"duration\": 9595}}",
  "input": {
    "type": "log"
  },
  "json_string": "{\"timestamp\": \"2019-06-04T10:15:23.174830-0300\", \"type\": \"Authentication\", \"Authentication\": {\"version\": {\"major\": 1, \"minor\": 0}, \"status\": \"NT_STATUS_OK\", \"localAddress\": null, \"remoteAddress\": \"ipv4:ip:port\", \"serviceDescription\": \"Kerberos KDC\", \"authDescription\": \"ENC-TS Pre-authentication\", \"clientDomain\": null, \"clientAccount\": \"account@domain\", \"workstation\": null, \"becameAccount\": \"account\", \"becameDomain\": \"DOMAIN\", \"becameSid\": \"S-1-5-21-310888293-2901519497-23123456789-1113\", \"mappedAccount\": \"account\", \"mappedDomain\": \"DOMAIN\", \"netlogonComputer\": null, \"netlogonTrustAccount\": null, \"netlogonNegotiateFlags\": \"0x00000000\", \"netlogonSecureChannelType\": 0, \"netlogonTrustAccountSid\": null, \"passwordType\": \"aes256-cts-hmac-sha1-96\", \"duration\": 9595}}",
  "json": {
    "Authentication": {
      "status": "NT_STATUS_OK",
      "remoteAddress": "ipv4:ip:port",
      "authDescription": "ENC-TS Pre-authentication",
      "netlogonSecureChannelType": 0,
      "netlogonTrustAccountSid": null,
      "serviceDescription": "Kerberos KDC",
      "becameAccount": "account",
      "duration": 9595,
      "netlogonTrustAccount": null,
      "version": {
        "major": 1,
        "minor": 0
      },
      "clientDomain": null,
      "becameDomain": "DOMAIN",
      "becameSid": "S-1-5-21-310888293-2901519497-23123456789-1113",
      "mappedAccount": "account",
      "mappedDomain": "DOMAIN",
      "netlogonComputer": null,
      "localAddress": null,
      "clientAccount": "account@domain",
      "workstation": null,
      "netlogonNegotiateFlags": "0x00000000",
      "passwordType": "aes256-cts-hmac-sha1-96"
    },
    "timestamp": "2019-06-04T10:15:23.174830-0300",
    "type": "Authentication"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "44eb8f3d7558"
  }
}

Valid variation of the tokenizer:

tokenizer: "%{}: %{json_string}"

Refs:
https://www.elastic.co/guide/en/beats/filebeat/7.0/dissect.html
https://www.elastic.co/guide/en/beats/filebeat/7.0/decode-json-fields.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html

elastic/beats/blob/master/libbeat/processors/dissect/testdata/dissect_tests.json

elastic/dissect-specification

Note 1:
I'm not saying the dissect processor in Beats is exactly like the logstash dissect filter.
As far as me reading the doc, source codes, the tests in the sources, github, the spec, they aren't.
But the filebeat dissect processor documentation page is thin and not up to date currently so you have to fall back on the logstash doc, the spec, the tests and the source codes to obtain knowledge and corrections to that knowledge when required. Of course simply empirically testing the hell out of the thing also works but you can't test for something you don't know is a feature...

Note 2:
There is a max_depth setting for the decode_json_fields processor. In your case it doesn't kick in and doesn't bound the deptness of the json object. I have experimented with different cases and I can make it kick in when I ingest json lines which contains escaped json strings in string fields. But even after reading the code and the github issues and forum posts I could find. I still don't quite understand it enough to explain it to someone else, logically. So I'll refrain from discussing this aspect at this point. I'm a bit confused by that setting currently.
Again it doesn't apply to your example, whatever you set it to.