I'm using filebeat 7.8.0 inside my kubernetes cluster.
With that configuration
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: logging
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
logging.level: warning
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
exclude_lines: '^[[:space:]]*$'
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
processors:
- add_cloud_metadata:
- add_host_metadata:
- decode_json_fields:
fields: ["message"]
target: "mycustomfield"
overwrite_keys: true
- add_fields:
target: ''
fields:
gkeclustername: core-prod
environment: prod
setup.ilm.policy_name: gke_prod_core
setup.template.settings:
index.number_of_shards: 2
index.number_of_replicas: 1
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
setup.ilm.rollover_alias: "${INDEX_NAME_CLUSTER}"
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
But I'm finding a lots of errors
filebeat 2020-07-31T07:46:21.910Z WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x2c3a8733, ext:63731778378, loc:(*time.Location)(nil)},
(bla bla bla)
"stream":"stdout"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc00049c340), Source:"/var/log/containers/mar-resource-proxy-df55b7b45-28g7s_default_mar-resource-proxy-c760cd4083fb33039cfcf27982e8d2626a82747503237dc1d36c1873071bcbb1.log", Offset:7921791, Timestamp:time.Time{wall:0xbfc11091918947dc, ext:420647607, loc:(*time.Location)(0x5b230e0)}, TTL:-1, Type:"container", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x11f684, Device:0x801}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [mycustomfield.target] of type [keyword] in document with id 'OFbWo3MBQyhZl6BJEDFJ'. Preview of field's value: '{resourceID=company-clique-plugin, actionPath=/api/blocks?returnTransactions=true&returnReceipts=true&height=8558692, requestID=a5aff329-f992-4fc8-b067-e5143d85f768}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1974"}}
Any idea where is the problem?
Maybe filebeat can't parser all the fields message as a Json? I thought that if It couldn't, nothing would happen, maybe one error, but the problem I am seeing is that it stops sending logs after a while.
I don't know if I can define for each application in kubernetes that I know the message is a json activate in some way this for that specific application.
- decode_json_fields:
fields: ["message"]
target: "mycustomfield"
overwrite_keys: true
Maybe with annotations in deployment of my service?
Any suggestion?
Thank you very much