Its possible activate the parser a json format for specific pod?

[David_Oceans]

I'm using filebeat 7.8.0 inside my kubernetes cluster.

With that configuration

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    logging.level: warning
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            exclude_lines: '^[[:space:]]*$'
            multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
            multiline.negate: false
            multiline.match: after
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - decode_json_fields:
          fields: ["message"]
          target: "mycustomfield"
          overwrite_keys: true          
      - add_fields:
          target: ''
          fields:
            gkeclustername: core-prod
            environment: prod

    setup.ilm.policy_name: gke_prod_core
    setup.template.settings:
      index.number_of_shards: 2
      index.number_of_replicas: 1

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    setup.ilm.rollover_alias: "${INDEX_NAME_CLUSTER}"

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

But I'm finding a lots of errors

filebeat 2020-07-31T07:46:21.910Z	WARN	[elasticsearch]	elasticsearch/client.go:407	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x2c3a8733, ext:63731778378, loc:(*time.Location)(nil)},  
(bla bla bla)
 "stream":"stdout"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc00049c340), Source:"/var/log/containers/mar-resource-proxy-df55b7b45-28g7s_default_mar-resource-proxy-c760cd4083fb33039cfcf27982e8d2626a82747503237dc1d36c1873071bcbb1.log", Offset:7921791, Timestamp:time.Time{wall:0xbfc11091918947dc, ext:420647607, loc:(*time.Location)(0x5b230e0)}, TTL:-1, Type:"container", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x11f684, Device:0x801}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [mycustomfield.target] of type [keyword] in document with id 'OFbWo3MBQyhZl6BJEDFJ'. Preview of field's value: '{resourceID=company-clique-plugin, actionPath=/api/blocks?returnTransactions=true&returnReceipts=true&height=8558692, requestID=a5aff329-f992-4fc8-b067-e5143d85f768}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1974"}}

Any idea where is the problem?

Maybe filebeat can't parser all the fields message as a Json? I thought that if It couldn't, nothing would happen, maybe one error, but the problem I am seeing is that it stops sending logs after a while.

I don't know if I can define for each application in kubernetes that I know the message is a json activate in some way this for that specific application.

      - decode_json_fields:
          fields: ["message"]
          target: "mycustomfield"
          overwrite_keys: true     

Maybe with annotations in deployment of my service?

Any suggestion?

Thank you very much

[David_Oceans]

Hi!

Its possible add this decode_json_fields at annotation level in each pod?

I only found this

co.elastic.logs/json.keys_under_root: true
co.elastic.logs/json.add_error_key: true
co.elastic.logs/json.message_key: message

But I don't think it's the same

I don't found any more information in docs about that

Thank you

[David_Oceans]

Any idea where can I go?

Thank you