Filebeat failed to parse JSON with nested object

Elastic Stack Beats
1

Hi,

I am trying to configure Filebeat to parse json logs produced by one of my service. Everything works find if log message has one level properties only. But if I include custom payload(nested object) Filebeat always fails to parse it. I have already processed these log message via Logstash without any problem but want to migrate to Filebeat.

filebeat.yml
filebeat.inputs:

- type: log
  enabled: true
  paths:
    - e:\Service\logs\*
  json.keys_under_root: true
  json.add_error_key: true

#processors:
#  - decode_json_fields:
#      fields: ["jsonmessage"]
#      process_array: true
#      max_depth: 6
#      overwrite_keys: true
#      when:
#        regexp:
#          jsonmessage: '^\{'
  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.enabled: false
setup.template.settings:
  index.number_of_shards: 3

setup.kibana:

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "logs-%{[beat.version]}-%{+yyyy.MM}"
  indices:
    - index: "%{[logindexprefix]}-%{+yyyy.MM}"
      when.regexp:
        logindexprefix: .*

log record example
{"date":"2018-07-16T18:02:22.4913711+01:00","level":"INFO","logger":"TcpServer","thread":"7","ndc":"(null)","message":"TEST MESSAGE","appname":"Service.DEVTEST","appversion":"1.1.0.6478","executable":"Service.exe","logindexprefix":"myservice","context":"SERVICE","method":"Run","connection":"(null)","imei":"(null)","payload":{"id":2,"msg":"Test message 2"}}

I've tried to POST above json directly to the index by using Kibana Dev Tools and it works.

Filebeat output:
2018-07-16T18:03:33.748+0300 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbecb4ad11141ea54, ext:133830301, loc:(*time.Location)(0x16f3580)}, Meta:common.MapStr(nil), Fields:common.MapStr{"host":common.MapStr{"name":"DEVELOPER"}, "executable":"Service.exe", "payload":common.MapStr{"id":1, "msg":"Test message"}, "message":"TEST MESSAGE", "logindexprefix":"myservice", "context":"SERVICE", "logger":"TcpServer", "appversion":"1.1.0.6478", "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "beat":common.MapStr{"name":"DEVELOPER", "hostname":"DEVELOPER", "version":"6.3.1"}, "imei":"(null)", "date":"2018-07-16T18:00:52.6016467+03:00", "offset":290222, "connection":"(null)", "ndc":"(null)", "source":"e:\\Service\\logs\\logfile", "appname":"Service.DEVTEST", "level":"INFO", "method":"Run", "thread":"8"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc042432a80), Source:"e:\\Service\\logs\\logfile", Offset:290637, Timestamp:time.Time{wall:0xbecb4ad1105a6530, ext:118657401, loc:(*time.Location)(0x16f3580)}, TTL:-1, Type:"log", Meta:map[string]string{}, FileStateOS:file.StateOS{IdxHi:0x1f0000, IdxLo:0x48a43, Vol:0x6e9748c}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [payload]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:593"}}

I also tried to serialize nested object to "jsonmessage" field and parse it via processor(commented lines in .yml file) but without any success.

Any help would be appreciated.

2

I don't get any errors indexing the example record you posted. Can you find a record that causes this error?

3

Thank you for the reply.

I included it as a "log record example".
Have you tested it with my Filebeat configuration?

4

Yes, used your config and fed the example event into ES without any trouble

5

Ok, got it. Thank you.
Probably I have to try different installation.

But what is the general meaning of error message: {"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:593"} ?

6

That is usually the error you get when parsing bad JSON.

Now that I look closely, in this case it may be that it's trying to parse the "message" field (value "TEST MESSAGE") as JSON. But the event already contains fields that are the result of JSON parsing, so it looks like double parsing to me.

Maybe it's the result of using both the json input and then the decode_json_fields processor?

7

I found the issue.

The records are parsed but are not written to the index because of existing index pattern where "payload" field has string type.

Thank you for your help.

9

I am trying to configure Filebeat to parse json logs produced by one of my service. Filebeat is not parsing the json object with separate fields and values. below is the config file I am using and the sampel document I am shipping via tcp.

filebeat.inputs:

  • type: tcp
    enabled: true
    max_message_size: 10MiB
    host: "ip:port"
    json.keys_under_root: true
    json.add_error_key: true

My output document,
{
"_index": "rackn-rdn-2018.08.09",
"_type": "doc",
"_id": "AWUgDV5ly79uHpPv6au7",
"_score": 1,
"_source": {
"@timestamp": "2018-08-09T18:57:51.358Z",
"host": {
"name": "host1"
},
"source": "30.130.41.59:45596",
"message": "{"Time":"2018-08-09T14:57:51.357952529-04:00","Type":"log","Action":"warn","Key":"frontend","Object":{"Data":[],"File":"/home/travis/gopath/src/github.com/digitalrebar/provision/frontend/frontend.go","Group":560386,"IgnorePublish":false,"Level":"warn","Line":287,"Message":"No DRP authentication token from xx.xxx.xx.xxx","Seq":271808,"Service":"frontend","Time":"2018-08-09T14:57:51.357950689-04:00"}}",
"prospector": {
"type": "tcp"
},
"input": {
"type": "tcp"
},
"beat": {
"name": "host1",
"hostname": "host1",
"version": "6.3.2"
}
},
"fields": {
"@timestamp": [
1533841071358
]
}
}

10

The tcp input doesn't have json parsing capabilities. json.keys_under_root and json.add_error_keys are valid only with the log input.

In your case, you need to use the Decode JSON fields processor:

processors:
  - decode_json_fields:
      fields: ['message']

(this configuration needs to be added to the top level of your filebeat.yml, not inside the tcp input)

2018-08-10T09:25:39.295+0200	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2018-08-10T07:25:39.295Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "message": {
    "Time": "2018-08-09T14:57:51.357952529-04:00",
    "Type": "log",
    "Action": "warn",
    "Key": "frontend",
    "Object": {
      "Data": [],
      "File": "/home/travis/gopath/src/github.com/digitalrebar/provision/frontend/frontend.go",
      "Group": 560386,
      "IgnorePublish": false,
      "Line": 287,
      "Time": "2018-08-09T14:57:51.357950689-04:00",
      "Level": "warn",
      "Message": "No DRP authentication token from xx.xxx.xx.xxx",
      "Seq": 271808,
      "Service": "frontend"
    }
  },
  "source": "127.0.0.1:58211",
  "prospector": {
    "type": "tcp"
  },
  "input": {
    "type": "tcp"
  },
  "beat": {
    "name": "Adrian-Elastic.local",
    "hostname": "Adrian-Elastic.local",
    "version": "7.0.0-alpha1"
  },
  "host": {
    "name": "Adrian-Elastic.local"
  }
}

Also, if you want the fields extracted to the top level event, not nested inside the message field, you need to add:

processors:
  - decode_json_fields:
      fields: ['message']
      target: ""
11

Thanks alot.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.