delphi
(Luis Mauricio)
October 25, 2019, 7:25pm
1
Hi.
I'm using Filebeat 7.4.0 and Google Pub/Sub as an input for reading Google App Engine exported logs.
Although the processor "decode_json_fields" is working fine, I'm getting an issue with nested fields not mapped correctly. Even after set mapping mannualy, the nested field (protoPayload.line) isn't working as expected.
The log example is https://pastebin.com/yqX9s6Bt .
Thanks.
kvch
(Noémi Ványi)
October 28, 2019, 6:04pm
2
Could you please share your Filebeat configuration and debug logs?
delphi
(Luis Mauricio)
October 29, 2019, 2:26pm
3
- type: google-pubsub
project_id: kubernetes-asdasdasd
topic: elastic-stack-asdasdasd
subscription.name: elastic-stack-asdasdasd-subscription
subscription.create: false
credentials_file: /usr/share/filebeat/pub-sub-key/elasticsearch-pubsub-k8s-prod.json
processors:
- decode_json_fields:
process_array: true
max_depth: 20
target: ""
overwrite_keys: true
fields: ["message"]
- drop_fields:
fields: ["message"]
- add_fields:
target: ''
fields:
topic_name: elastic-stack-gae-asdasdas
monitoring.enabled: "true"
monitoring.elasticsearch.username: ${beats-username}
monitoring.elasticsearch.password: ${beats-password}
queue.mem:
events: 4096
setup.dashboards.enabled: false
setup.template:
enabled: true
overwrite: true
name: flb-k8s
pattern: "flb-k8s-*"
settings.index:
number_of_shards: 3
number_of_replicas: 0
number_of_routing_shards: 30
routing.allocation.require.node_type: "hot"
output.elasticsearch:
worker: 2
hosts: http://elastic-production-coordinating:9200
username: ${filebeat-elastic-username}
password: ${filebeat-elastic-password}
indices:
- index: "flb-k8s-pubsub-%{[topic_name]}"
when.contains:
input.type: "google-pubsub"
kvch
(Noémi Ványi)
October 29, 2019, 3:05pm
4
Could you also share debug logs? Or at least the beat events this configuration produces?
delphi
(Luis Mauricio)
October 29, 2019, 8:38pm
5
Beats-events == filebeat log? Or a Elasticsearch document?
system
(system)
Closed
November 26, 2019, 8:38pm
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.