# Filebeat failed to parse JSON with nested object **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 16, 2018, 7:36pm UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209 "2018-07-16T19:36:54Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![grigory](https://avatars.discourse-cdn.com/v4/letter/g/a587f6/32.png) [@grigory](https://discuss.elastic.co/u/grigory)\ **Post date:** [July 16, 2018, 7:36pm UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/1 "2018-07-16T19:36:54Z")
Hi, I am trying to configure Filebeat to parse json logs produced by one of my service. Everything works find if log message has one level properties only. But if I include custom payload(nested object) Filebeat always fails to parse it. I have already processed these log message via Logstash without any problem but want to migrate to Filebeat. filebeat.yml filebeat.inputs: ``` - type: log enabled: true paths: - e:\Service\logs\* json.keys_under_root: true json.add_error_key: true #processors: # - decode_json_fields: # fields: ["jsonmessage"] # process_array: true # max_depth: 6 # overwrite_keys: true # when: # regexp: # jsonmessage: '^\{' filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.enabled: false setup.template.settings: index.number_of_shards: 3 setup.kibana: output.elasticsearch: hosts: ["localhost:9200"] index: "logs-%{[beat.version]}-%{+yyyy.MM}" indices: - index: "%{[logindexprefix]}-%{+yyyy.MM}" when.regexp: logindexprefix: .* ``` log record example `{"date":"2018-07-16T18:02:22.4913711+01:00","level":"INFO","logger":"TcpServer","thread":"7","ndc":"(null)","message":"TEST MESSAGE","appname":"Service.DEVTEST","appversion":"1.1.0.6478","executable":"Service.exe","logindexprefix":"myservice","context":"SERVICE","method":"Run","connection":"(null)","imei":"(null)","payload":{"id":2,"msg":"Test message 2"}}` I've tried to POST above json directly to the index by using Kibana Dev Tools and it works. Filebeat output: `2018-07-16T18:03:33.748+0300 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbecb4ad11141ea54, ext:133830301, loc:(*time.Location)(0x16f3580)}, Meta:common.MapStr(nil), Fields:common.MapStr{"host":common.MapStr{"name":"DEVELOPER"}, "executable":"Service.exe", "payload":common.MapStr{"id":1, "msg":"Test message"}, "message":"TEST MESSAGE", "logindexprefix":"myservice", "context":"SERVICE", "logger":"TcpServer", "appversion":"1.1.0.6478", "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "beat":common.MapStr{"name":"DEVELOPER", "hostname":"DEVELOPER", "version":"6.3.1"}, "imei":"(null)", "date":"2018-07-16T18:00:52.6016467+03:00", "offset":290222, "connection":"(null)", "ndc":"(null)", "source":"e:\\Service\\logs\\logfile", "appname":"Service.DEVTEST", "level":"INFO", "method":"Run", "thread":"8"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc042432a80), Source:"e:\\Service\\logs\\logfile", Offset:290637, Timestamp:time.Time{wall:0xbecb4ad1105a6530, ext:118657401, loc:(*time.Location)(0x16f3580)}, TTL:-1, Type:"log", Meta:map[string]string{}, FileStateOS:file.StateOS{IdxHi:0x1f0000, IdxLo:0x48a43, Vol:0x6e9748c}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [payload]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:593"}}` I also tried to serialize nested object to "jsonmessage" field and parse it via processor(commented lines in .yml file) but without any success. Any help would be appreciated. ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [July 17, 2018, 8:37am UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/2 "2018-07-17T08:37:36Z")
I don't get any errors indexing the example record you posted. Can you find a record that causes this error? ---
**Author:** ![grigory](https://avatars.discourse-cdn.com/v4/letter/g/a587f6/32.png) [@grigory](https://discuss.elastic.co/u/grigory)\ **Post date:** [July 17, 2018, 9:56am UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/3 "2018-07-17T09:56:35Z")
Thank you for the reply. I included it as a "log record example". Have you tested it with my Filebeat configuration? ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [July 17, 2018, 10:09am UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/4 "2018-07-17T10:09:54Z")
Yes, used your config and fed the example event into ES without any trouble ---
**Author:** ![grigory](https://avatars.discourse-cdn.com/v4/letter/g/a587f6/32.png) [@grigory](https://discuss.elastic.co/u/grigory)\ **Post date:** [July 17, 2018, 10:17am UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/5 "2018-07-17T10:17:02Z")
Ok, got it. Thank you. Probably I have to try different installation. But what is the general meaning of error message: {"type":"illegal\_state\_exception","reason":"Can't get text on a START\_OBJECT at 1:593"} ? ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [July 17, 2018, 10:38am UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/6 "2018-07-17T10:38:44Z")
That is usually the error you get when parsing bad JSON. Now that I look closely, in this case it may be that it's trying to parse the "message" field (value "TEST MESSAGE") as JSON. But the event already contains fields that are the result of JSON parsing, so it looks like double parsing to me. Maybe it's the result of using both the json input and then the decode\_json\_fields processor? ---
**Author:** ![grigory](https://avatars.discourse-cdn.com/v4/letter/g/a587f6/32.png) [@grigory](https://discuss.elastic.co/u/grigory)\ **Post date:** [July 17, 2018, 10:52am UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/7 "2018-07-17T10:52:31Z")
I found the issue. The records are parsed but are not written to the index because of existing index pattern where "payload" field has string type. Thank you for your help. ---
**Author:** ![chakri276](https://avatars.discourse-cdn.com/v4/letter/c/e99b99/32.png) [@chakri276](https://discuss.elastic.co/u/chakri276)\ **Post date:** [August 9, 2018, 8:16pm UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/9 "2018-08-09T20:16:43Z")
I am trying to configure Filebeat to parse json logs produced by one of my service. Filebeat is not parsing the json object with separate fields and values. below is the config file I am using and the sampel document I am shipping via tcp. filebeat.inputs: - type: tcp enabled: true max\_message\_size: 10MiB host: "ip:port" json.keys\_under\_root: true json.add\_error\_key: true My output document, { "\_index": "rackn-rdn-2018.08.09", "\_type": "doc", "\_id": "AWUgDV5ly79uHpPv6au7", "\_score": 1, "\_source": { "@timestamp": "2018-08-09T18:57:51.358Z", "host": { "name": "[host1](http://host2/)" }, "source": "30.130.41.59:45596", "message": "{"Time":"2018-08-09T14:57:51.357952529-04:00","Type":"log","Action":"warn","Key":"frontend","Object":{"Data":[],"File":"/home/travis/gopath/src/github.com/digitalrebar/provision/frontend/frontend.go","Group":560386,"IgnorePublish":false,"Level":"warn","Line":287,"Message":"No DRP authentication token from xx.xxx.xx.xxx","Seq":271808,"Service":"frontend","Time":"2018-08-09T14:57:51.357950689-04:00"}}", "prospector": { "type": "tcp" }, "input": { "type": "tcp" }, "beat": { "name": "[host1](http://host1/)", "hostname": "[host1](http://host2/)", "version": "6.3.2" } }, "fields": { "@timestamp": [ 1533841071358 ] } } ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [August 10, 2018, 7:30am UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/10 "2018-08-10T07:30:36Z")
The `tcp` input doesn't have `json` parsing capabilities. `json.keys_under_root` and `json.add_error_keys` are valid only with the `log` input. In your case, you need to use the [Decode JSON fields processor](https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html): ```auto processors: - decode_json_fields: fields: ['message'] ``` (this configuration needs to be added to the top level of your `filebeat.yml`, not inside the tcp input) ```auto 2018-08-10T09:25:39.295+0200 DEBUG [publish] pipeline/processor.go:308 Publish event: { "@timestamp": "2018-08-10T07:25:39.295Z", "@metadata": { "beat": "filebeat", "type": "doc", "version": "7.0.0-alpha1" }, "message": { "Time": "2018-08-09T14:57:51.357952529-04:00", "Type": "log", "Action": "warn", "Key": "frontend", "Object": { "Data": [], "File": "/home/travis/gopath/src/github.com/digitalrebar/provision/frontend/frontend.go", "Group": 560386, "IgnorePublish": false, "Line": 287, "Time": "2018-08-09T14:57:51.357950689-04:00", "Level": "warn", "Message": "No DRP authentication token from xx.xxx.xx.xxx", "Seq": 271808, "Service": "frontend" } }, "source": "127.0.0.1:58211", "prospector": { "type": "tcp" }, "input": { "type": "tcp" }, "beat": { "name": "Adrian-Elastic.local", "hostname": "Adrian-Elastic.local", "version": "7.0.0-alpha1" }, "host": { "name": "Adrian-Elastic.local" } } ``` Also, if you want the fields extracted to the top level event, not nested inside the `message` field, you need to add: ```auto processors: - decode_json_fields: fields: ['message'] target: "" ``` ---
**Author:** ![chakri276](https://avatars.discourse-cdn.com/v4/letter/c/e99b99/32.png) [@chakri276](https://discuss.elastic.co/u/chakri276)\ **Post date:** [August 12, 2018, 10:17pm UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/11 "2018-08-12T22:17:18Z")
> [@adrisr](#): > > ```auto > "type": "tcp" > }, > "input": { > "type": "tcp" > }, > "beat": { > "name": "Adrian-Elastic.local", > "hostname": "Adrian-Elastic.local", > "version": "7.0.0-alpha1" > }, > "host": { > "name": "Adrian-Elastic.local" > } > } > > ``` > > Also, if you want the fields extracted to the top level eve Thanks alot. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 9, 2018, 10:17pm UTC](https://discuss.elastic.co/t/filebeat-failed-to-parse-json-with-nested-object/140209/12 "2018-09-09T22:17:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.