# Filebeat "decode\_json\_fields" and nested fields

**URL:** <https://discuss.elastic.co/t/filebeat-decode-json-fields-and-nested-fields/205316>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 25, 2019, 7:25pm UTC](https://discuss.elastic.co/t/filebeat-decode-json-fields-and-nested-fields/205316 "2019-10-25T19:25:24Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![delphi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/delphi/32/52397_2.png) [@delphi](https://discuss.elastic.co/u/delphi)\
**Post date:** [October 25, 2019, 7:25pm UTC](https://discuss.elastic.co/t/filebeat-decode-json-fields-and-nested-fields/205316/1 "2019-10-25T19:25:24Z")

</div>

Hi.

I'm using Filebeat 7.4.0 and Google Pub/Sub as an input for reading Google App Engine exported logs.

Although the processor "decode\_json\_fields" is working fine, I'm getting an issue with nested fields not mapped correctly. Even after set mapping mannualy, the nested field (protoPayload.line) isn't working as expected.

The log example is [https://pastebin.com/yqX9s6Bt](https://pastebin.com/yqX9s6Bt) .

Thanks.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 28, 2019, 6:04pm UTC](https://discuss.elastic.co/t/filebeat-decode-json-fields-and-nested-fields/205316/2 "2019-10-28T18:04:38Z")

</div>

Could you please share your Filebeat configuration and debug logs?

---

<div class="post-metadata">

**Author:** ![delphi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/delphi/32/52397_2.png) [@delphi](https://discuss.elastic.co/u/delphi)\
**Post date:** [October 29, 2019, 2:26pm UTC](https://discuss.elastic.co/t/filebeat-decode-json-fields-and-nested-fields/205316/3 "2019-10-29T14:26:37Z")

</div>

```
- type: google-pubsub
  project_id: kubernetes-asdasdasd
  topic: elastic-stack-asdasdasd
  subscription.name: elastic-stack-asdasdasd-subscription
  subscription.create: false
  credentials_file: /usr/share/filebeat/pub-sub-key/elasticsearch-pubsub-k8s-prod.json
  processors:
    - decode_json_fields:
        process_array: true
        max_depth: 20
        target: ""
        overwrite_keys: true
        fields: ["message"]
    - drop_fields:
        fields: ["message"]
    - add_fields:
        target: ''
        fields:
          topic_name: elastic-stack-gae-asdasdas

monitoring.enabled: "true"
monitoring.elasticsearch.username: ${beats-username}
monitoring.elasticsearch.password: ${beats-password}

queue.mem:
  events: 4096

setup.dashboards.enabled: false
setup.template:
  enabled: true
  overwrite: true
  name: flb-k8s
  pattern: "flb-k8s-*"
  settings.index:
    number_of_shards: 3
    number_of_replicas: 0
    number_of_routing_shards: 30
    routing.allocation.require.node_type: "hot"

output.elasticsearch:
  worker: 2
  hosts: http://elastic-production-coordinating:9200
  username: ${filebeat-elastic-username}
  password: ${filebeat-elastic-password}
  indices:
    - index: "flb-k8s-pubsub-%{[topic_name]}"
      when.contains:
        input.type: "google-pubsub"
```

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [October 29, 2019, 3:05pm UTC](https://discuss.elastic.co/t/filebeat-decode-json-fields-and-nested-fields/205316/4 "2019-10-29T15:05:32Z")

</div>

Could you also share debug logs? Or at least the beat events this configuration produces?

---

<div class="post-metadata">

**Author:** ![delphi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/delphi/32/52397_2.png) [@delphi](https://discuss.elastic.co/u/delphi)\
**Post date:** [October 29, 2019, 8:38pm UTC](https://discuss.elastic.co/t/filebeat-decode-json-fields-and-nested-fields/205316/5 "2019-10-29T20:38:05Z")

</div>

Beats-events == filebeat log? Or a Elasticsearch document?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 26, 2019, 8:38pm UTC](https://discuss.elastic.co/t/filebeat-decode-json-fields-and-nested-fields/205316/6 "2019-11-26T20:38:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.