# Unable to push nested json from filebeat to elasticsearch

**URL:** <https://discuss.elastic.co/t/unable-to-push-nested-json-from-filebeat-to-elasticsearch/194920>\
**Category:** Elasticsearch\
**Tags:** beats-module\
**Created:** [August 12, 2019, 8:52pm UTC](https://discuss.elastic.co/t/unable-to-push-nested-json-from-filebeat-to-elasticsearch/194920 "2019-08-12T20:52:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kumar\_Pratyush](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kumar_pratyush/32/52133_2.png) [@Kumar\_Pratyush](https://discuss.elastic.co/u/Kumar_Pratyush)\
**Post date:** [August 12, 2019, 8:52pm UTC](https://discuss.elastic.co/t/unable-to-push-nested-json-from-filebeat-to-elasticsearch/194920/1 "2019-08-12T20:52:03Z")

</div>

So my requirement is fairly simple. I have logs in the form of json where each line corresponds to one document. The json is in nested format. When I use a simple json, i.e. without any nestings, it works fine and the same is pushed to Elasticsearch as well. However, the nested json throws error.  
My stack details :  
**Elastic stack** : 7.2.0 (Docker container)  
**Filebeats** : 7.3.0 (local setup, non-docker)

My filebeats config file is :

```
filebeat.inputs:
- type: log
  paths:
  - /Users/quiqua/Downloads/test.json
  json.keys_under_root: true
  json.add_error_key: true
processors:
- decode_json_fields:
    fields: ["unittests"]
    process_array: true
output.elasticsearch:
  hosts: ["localhost:9200"]

```

My log file :

> {"test":"works"}  
> {"test":{"nested":"this doesn't work"}}

Error trace :

> 2019-08-13T02:23:15.082+0530 WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf4c924e8416e7d0, ext:190176101272, loc:(\*time.Location)(0x62bc980)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral\_id":"78cf1fd6-d809-4bb2-8f91-c5020a3baa67", "hostname":"quiQUAs-MacBook-Pro.local", "id":"17690335-33e8-4690-b8e0-308062e40507", "type":"filebeat", "version":"7.3.0"}, "ecs":common.MapStr{"version":"1.0.1"}, "host":common.MapStr{"name":"quiQUAs-MacBook-Pro.local"}, "input":common.MapStr{"type":"log"}, "log":common.MapStr{"file":common.MapStr{"path":"/Users/quiqua/Downloads/test.json"}, "offset":17}, "test":common.MapStr{"nested":"this doesn't work"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(\*os.fileStat)(0xc0003241a0), Source:"/Users/quiqua/Downloads/test.json", Offset:57, Timestamp:time.Time{wall:0xbf4c924e840b2418, ext:190175330707, loc:(\*time.Location)(0x62bc980)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xa18955, Device:0x1000007}}, TimeSeries:false}, Flags:0x1} (status=400): {"type":"mapper\_parsing\_exception","reason":"failed to parse field [test] of type [keyword] in document with id 'Ajqbh2wBbkh2UY59QjrF'","caused\_by":{"type":"illegal\_state\_exception","reason":"Can't get text on a START\_OBJECT at 1:49"}}

I have tried all possible solutions and yet the issue persists. Any help would be great. Thank you.

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [August 14, 2019, 10:47am UTC](https://discuss.elastic.co/t/unable-to-push-nested-json-from-filebeat-to-elasticsearch/194920/2 "2019-08-14T10:47:06Z")

</div>

Even though Elasticsearch is sometimes (incorrectly) called schemaless, it does in fact have a data schema, called "[mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html)". If you try to index a document that contains fields that have not been mapped, Elasticsearch will [dynamically map](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#_dynamic_mapping) those fields.

In your case, Filebeat ingests the first line into Elasticsearch which contains the `test` field that contains a string value. Elasticsearch will dynamically map that field as a ["`keyword`" string](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html).

Then, Filebeat tries to ingest the second line. This line also contains a `test` field, but that field contains a JSON object instead of a string. This now leads to a mapping conflict: Elasticsearch cannot index an object into a field that has already been mapped as a keyword string.

What's the solution? Give your fields a different name, depending on the type of value that they contain. The following file should work without any problems:

```auto
{"test":"works"}
{"test2":{"nested":"this does also work"}}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 11, 2019, 10:47am UTC](https://discuss.elastic.co/t/unable-to-push-nested-json-from-filebeat-to-elasticsearch/194920/3 "2019-09-11T10:47:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.