Filebeat "decode_json_fields" and nested fields

[delphi (Luis Mauricio)]

Hi.

I'm using Filebeat 7.4.0 and Google Pub/Sub as an input for reading Google App Engine exported logs.

Although the processor "decode_json_fields" is working fine, I'm getting an issue with nested fields not mapped correctly. Even after set mapping mannualy, the nested field (protoPayload.line) isn't working as expected.

The log example is https://pastebin.com/yqX9s6Bt .

Thanks.

[kvch (Noémi Ványi)]

Could you please share your Filebeat configuration and debug logs?

[delphi (Luis Mauricio)]

- type: google-pubsub
  project_id: kubernetes-asdasdasd
  topic: elastic-stack-asdasdasd
  subscription.name: elastic-stack-asdasdasd-subscription
  subscription.create: false
  credentials_file: /usr/share/filebeat/pub-sub-key/elasticsearch-pubsub-k8s-prod.json
  processors:
    - decode_json_fields:
        process_array: true
        max_depth: 20
        target: ""
        overwrite_keys: true
        fields: ["message"]
    - drop_fields:
        fields: ["message"]
    - add_fields:
        target: ''
        fields:
          topic_name: elastic-stack-gae-asdasdas

monitoring.enabled: "true"
monitoring.elasticsearch.username: ${beats-username}
monitoring.elasticsearch.password: ${beats-password}

queue.mem:
  events: 4096

setup.dashboards.enabled: false
setup.template:
  enabled: true
  overwrite: true
  name: flb-k8s
  pattern: "flb-k8s-*"
  settings.index:
    number_of_shards: 3
    number_of_replicas: 0
    number_of_routing_shards: 30
    routing.allocation.require.node_type: "hot"

output.elasticsearch:
  worker: 2
  hosts: http://elastic-production-coordinating:9200
  username: ${filebeat-elastic-username}
  password: ${filebeat-elastic-password}
  indices:
    - index: "flb-k8s-pubsub-%{[topic_name]}"
      when.contains:
        input.type: "google-pubsub"

[kvch (Noémi Ványi)]

Could you also share debug logs? Or at least the beat events this configuration produces?

[delphi (Luis Mauricio)]

Beats-events == filebeat log? Or a Elasticsearch document?