# Its possible activate the parser a json format for specific pod?

**URL:** https://discuss.elastic.co/t/its-possible-activate-the-parser-a-json-format-for-specific-pod/243325
**Category:** Beats
**Tags:** filebeat
**Created:** [July 31, 2020, 8:42am UTC](https://discuss.elastic.co/t/its-possible-activate-the-parser-a-json-format-for-specific-pod/243325 "2020-07-31T08:42:13Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![David\_Oceans](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_oceans/32/51452_2.png) [@David\_Oceans](https://discuss.elastic.co/u/David_Oceans)
#### Post date: [July 31, 2020, 8:42am UTC](https://discuss.elastic.co/t/its-possible-activate-the-parser-a-json-format-for-specific-pod/243325/1 "2020-07-31T08:42:13Z")

</div>

I'm using filebeat 7.8.0 inside my kubernetes cluster.

With that configuration

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    logging.level: warning
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            exclude_lines: '^[[:space:]]*$'
            multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
            multiline.negate: false
            multiline.match: after
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - decode_json_fields:
          fields: ["message"]
          target: "mycustomfield"
          overwrite_keys: true          
      - add_fields:
          target: ''
          fields:
            gkeclustername: core-prod
            environment: prod

    setup.ilm.policy_name: gke_prod_core
    setup.template.settings:
      index.number_of_shards: 2
      index.number_of_replicas: 1

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    setup.ilm.rollover_alias: "${INDEX_NAME_CLUSTER}"

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

```

But I'm finding a lots of errors

```auto
filebeat 2020-07-31T07:46:21.910Z	WARN	[elasticsearch]	elasticsearch/client.go:407	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x2c3a8733, ext:63731778378, loc:(*time.Location)(nil)},  
(bla bla bla)
 "stream":"stdout"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc00049c340), Source:"/var/log/containers/mar-resource-proxy-df55b7b45-28g7s_default_mar-resource-proxy-c760cd4083fb33039cfcf27982e8d2626a82747503237dc1d36c1873071bcbb1.log", Offset:7921791, Timestamp:time.Time{wall:0xbfc11091918947dc, ext:420647607, loc:(*time.Location)(0x5b230e0)}, TTL:-1, Type:"container", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x11f684, Device:0x801}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [mycustomfield.target] of type [keyword] in document with id 'OFbWo3MBQyhZl6BJEDFJ'. Preview of field's value: '{resourceID=company-clique-plugin, actionPath=/api/blocks?returnTransactions=true&returnReceipts=true&height=8558692, requestID=a5aff329-f992-4fc8-b067-e5143d85f768}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1974"}}

```

Any idea where is the problem?

Maybe filebeat can't parser all the fields message as a Json? I thought that if It couldn't, nothing would happen, maybe one error, but the problem I am seeing is that it stops sending logs after a while.

I don't know if I can define for each application in kubernetes that I know the message is a json activate in some way this for that specific application.

```auto
      - decode_json_fields:
          fields: ["message"]
          target: "mycustomfield"
          overwrite_keys: true     

```

Maybe with annotations in deployment of my service?

Any suggestion?

Thank you very much

---

<div class="post-metadata">

### Author: ![David\_Oceans](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_oceans/32/51452_2.png) [@David\_Oceans](https://discuss.elastic.co/u/David_Oceans)
#### Post date: [August 3, 2020, 1:06pm UTC](https://discuss.elastic.co/t/its-possible-activate-the-parser-a-json-format-for-specific-pod/243325/2 "2020-08-03T13:06:18Z")

</div>

Hi!

Its possible add this decode\_json\_fields at annotation level in each pod?

I only found this

```auto
co.elastic.logs/json.keys_under_root: true
co.elastic.logs/json.add_error_key: true
co.elastic.logs/json.message_key: message

```

But I don't think it's the same

I don't found any more information in docs about that 😢

Thank you

---

<div class="post-metadata">

### Author: ![David\_Oceans](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_oceans/32/51452_2.png) [@David\_Oceans](https://discuss.elastic.co/u/David_Oceans)
#### Post date: [August 6, 2020, 3:23pm UTC](https://discuss.elastic.co/t/its-possible-activate-the-parser-a-json-format-for-specific-pod/243325/3 "2020-08-06T15:23:38Z")

</div>

Any idea where can I go?

Thank you

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 3, 2020, 3:23pm UTC](https://discuss.elastic.co/t/its-possible-activate-the-parser-a-json-format-for-specific-pod/243325/4 "2020-09-03T15:23:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
