# Filebeat: Bug: Processor "decode\_json\_fields" with usage of "target"

**URL:** <https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 11, 2022, 12:18pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972 "2022-02-11T12:18:13Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![fwetzler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fwetzler/32/101720_2.png) [@fwetzler](https://discuss.elastic.co/u/fwetzler)\
**Post date:** [February 11, 2022, 12:18pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/1 "2022-02-11T12:18:13Z")

</div>

OS: SLES12, CentOS 7.9  
Filebeat Versions: 7.8.0, 7.16.2

After using processor "decode\_json\_fields" WITH "target: 'sometarget' it's impossible to access some extracted json fields with following processors.

If you let "target": '' (unset) the following processors are able to access the extracted json fields.

```auto
##############################
# running fine
  processors:
    - if:
        regexp:
          message: '"parameter\d+":\w+\.?\d*'
      then:
        - add_fields:
            target: ''
            fields:
              not_isy: Parameter_not_string

    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: ""
        overwrite_keys: false
        add_error_key: true
...
# This can extract from field "exception"
    - dissect:
        tokenizer: '%{exception_source}: #%{exception_code} %{exception_err_msg}. %{exception_rest}'
        field: exception
        target_prefix: ''

############################
# NOT running fine
  processors:
    - if:
        regexp:
          message: '"parameter\d+":\w+\.?\d*'
      then:
        - add_fields:
            target: ''
            fields:
              not_isy: Parameter_not_string

    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: "mytarget"
        overwrite_keys: false
        add_error_key: true
...
# Never doing anything
    - dissect:
        tokenizer: '%{exception_source}: #%{exception_code} %{exception_err_msg}. %{exception_rest}'
        field: mytarget_exception
        target_prefix: 'mytarget'

```

I have made every attempt I can imagine: e.g. "if has\_fields" cannot detect the field, but it's definitely present. Same with regex ...  
I tried e.g.:

```auto
# - if:
# has_fields: ['mytarget_exception']
# then:
# - add_fields:
# target: 'mytarget'
# fields:
# exception_found: yes

```

I also tried with "has\_fields: ['exception']" ... no success

Every time I worked with usage of "target" or "target\_prefix" it was failing ...  
Now I work without that and my 322-line-filebeat-config does (nearly) what I want.

There are some more issues like "multiline pattern only working with type "log" but NOT with type "filestream" and so on. But that in another issue ... 😉

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 11, 2022, 12:40pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/2 "2022-02-11T12:40:18Z")

</div>

To me, it seems like misconfiguration. You put the results of the JSON parsing to under the field `mytarget`, but in the `dissect` processor you try to parse the field `mytarget_exception` which is I assume is empty.

But it would be nice to see an example event you are trying to parse.

> There are some more issues like "multiline pattern only working with type "log" but NOT with type "filestream" and so on. But that in another issue ... 😉

Then please post an another question... 😉

---

<div class="post-metadata">

**Author:** ![fwetzler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fwetzler/32/101720_2.png) [@fwetzler](https://discuss.elastic.co/u/fwetzler)\
**Post date:** [February 11, 2022, 12:50pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/3 "2022-02-11T12:50:37Z")

</div>

Hi,  
"decode\_json\_fields" extracts the field "exception" which I want to deal with other processors.  
This works only WITHOUT setting "target" to something else than empty (''). Please see my 1st code-example with "# running fine" comment.

Unfortunately I must not give a real life example.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 11, 2022, 1:00pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/4 "2022-02-11T13:00:31Z")

</div>

If you set the `target` of `decode_json_fields` to an empty value, Filebeat puts the fields to the root of the event. I assume one of the parsed fields is called `exception`. Then in the later `dissect` processor, you configure it as the source, and it can be parsed as expected.

However, in your second configuration snippet that does not work you put the results of the JSON processor under `mytarget`. Your `expcetion` field lands under the name `mytarget.exception`. So you should configure `mytarget.exception` as the `field` of `dissect` processor. I assume that is the field you would like to dissect.

---

<div class="post-metadata">

**Author:** ![fwetzler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fwetzler/32/101720_2.png) [@fwetzler](https://discuss.elastic.co/u/fwetzler)\
**Post date:** [February 11, 2022, 1:15pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/5 "2022-02-11T13:15:32Z")

</div>

I tried that, see 2nd example with comment "# never doing anything"

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 11, 2022, 1:35pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/6 "2022-02-11T13:35:41Z")

</div>

No, in your example you have `mytarget_exception`, not `mytarget.exception`.

---

<div class="post-metadata">

**Author:** ![fwetzler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fwetzler/32/101720_2.png) [@fwetzler](https://discuss.elastic.co/u/fwetzler)\
**Post date:** [February 11, 2022, 1:38pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/7 "2022-02-11T13:38:06Z")

</div>

I also tried that. "mytarget.exception" was my last try ... sorry for beeing not so clear!

---

<div class="post-metadata">

**Author:** ![fwetzler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fwetzler/32/101720_2.png) [@fwetzler](https://discuss.elastic.co/u/fwetzler)\
**Post date:** [February 11, 2022, 1:47pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/8 "2022-02-11T13:47:17Z")

</div>

I tried and analysed for appr. 8h, with filebeat debug logs, looking into Graylog for which fields where extracted, processed and so on.  
My "running" filebeat config (without any "target"-setting) has 322 lines, including multiline pattern in other inputs. So I assume to have understood most of the concept.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 11, 2022, 1:57pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/9 "2022-02-11T13:57:36Z")

</div>

Let's take a step back. What if you temporarily remove the dissect processor and write to file output to see where the fields from `decode_json_fields` land?

---

<div class="post-metadata">

**Author:** ![fwetzler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fwetzler/32/101720_2.png) [@fwetzler](https://discuss.elastic.co/u/fwetzler)\
**Post date:** [February 11, 2022, 2:20pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/10 "2022-02-11T14:20:18Z")

</div>

Scenario 1: target: "mytarget":  
If I look into Graylog, I can see the field "mytarget\_exception", also all other json-decoded fields. But in filebeat I cannot access, no processor can "see" this field(s) : no "IF hasfield", no "dissect" ,...

Scenario 2: target unset:  
All expected json\_decoded fields AND processed derivates of them are present in Graylog. So in Filebeat I can process json\_decoded fields with "dissect", I can process with "IF hasfield ..." ...

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 11, 2022, 2:31pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/11 "2022-02-11T14:31:01Z")

</div>

Are there any other processors or parsers besides the ones you shared? I am trying to track down where exactly the field is born and why it is not "visible" to the later processor.

---

<div class="post-metadata">

**Author:** ![fwetzler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fwetzler/32/101720_2.png) [@fwetzler](https://discuss.elastic.co/u/fwetzler)\
**Post date:** [February 11, 2022, 3:05pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/12 "2022-02-11T15:05:43Z")

</div>

Here is the complete section which is NOT running related to usage of "target" or "target\_prefix".

If you don't use "target" (and change appropriate fields) everything is fine (fields just have no prefix in Graylog)

```auto
#=========================== Filebeat inputs =============================

##### global options #####
processors:
  - add_host_metadata: ~

filebeat.inputs:
#####  
- type: log
  enabled: true
  fields_under_root: true
  ignore_older: 48h
  max_bytes: 32000
  scan_frequency: 10s

  paths:
    - /var/log/....

  fields:
    input_tag: plis
    target_index: stu_old_files
    env: sb-s2

  # not_isy correction
  processors:
    - if:
        regexp:
          message: '"parameter\d+":\w+\.?\d*'
      then:
        - add_fields:
            target: ''
            fields:
              not_isy: Parameter_not_string

    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: "plis"
        overwrite_keys: false
        add_error_key: true

    # Alle Felder: string-Format sicherstellen
    - convert:
        fields:
          - {from: "plis_parameter1", type: "string"}
          - {from: "plis_parameter2", type: "string"}
          - {from: "plis_parameter3", type: "string"}
          - {from: "plis_parameter4", type: "string"}
# test - {from: "parameter4", to: "parameter4_convert", type: "string"}
          - {from: "plis_parameter5", type: "string"}
          - {from: "plis_parameter6", type: "string"}
          - {from: "plis_parameter7", type: "string"}
          - {from: "plis_parameter8", type: "string"}
          - {from: "plis_parameter9", type: "string"}
          - {from: "plis_parameter10", type: "string"}
          - {from: "plis_parameter11", type: "string"}
          - {from: "plis_parameter12", type: "string"}
        ignore_missing: true
        fail_on_error: false
        mode: copy

# # application aus log_file_path
    - dissect:
        tokenizer: '/var/log/%{application}/%{log_file_name}'
        # not running: field: log_file_path
        field: log.file.path
        target_prefix: 'plis'

    # Exception code
    - if:
        regexp:
          plis_exception: ": #"
      then:
        - dissect:
## tokenizer: '%{plis_exception_source}: #%{plis_exception_code} %{plis_exception_err_msg}. %{plis_exception_prelude} Caused by: %{plis_exception_cause} at %{plis_exception_rest}'
            tokenizer: '%{exception_source}: #%{exception_code} %{exception_err_msg}. %{exception_rest}'
            field: plis_exception
            target_prefix: 'plis'
      else:
        - dissect:
            tokenizer: '%{exception_err_msg}: %{exception_rest}'
            field: plis_exception
            target_prefix: 'plis'

    # ggf. erzeugtes extract-Field wieder loeschen
    - drop_fields:
        fields: ["plis_exception_rest"]

```

---

<div class="post-metadata">

**Author:** ![fwetzler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fwetzler/32/101720_2.png) [@fwetzler](https://discuss.elastic.co/u/fwetzler)\
**Post date:** [February 15, 2022, 11:25am UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/13 "2022-02-15T11:25:57Z")

</div>

Hello Noémi, anything new on this issue?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 16, 2022, 5:07pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/14 "2022-02-16T17:07:17Z")

</div>

I tried to came up with an example event based on your configuration. I used this:

```auto
{"plis_exception": "my-source: #123 my expection message. more message"}

```

I changed the targets to `plis.plis_expection` in the exception handling `dissect` processors and got this event:

```auto
{
  "@timestamp": "2022-02-16T17:03:13.272Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.2.0"
  },
  "input_tag": "plis",
  "plis": {
    "exception_err_msg": "my expection message",
    "plis_exception": "my-source: #123 my expection message. more message",
    "application": "filebeat",
    "log_file_name": "test.log",
    "exception_rest": "more message",
    "exception_source": "my-source",
    "exception_code": "123"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "name": "sleipnir",
    "type": "filebeat",
    "version": "8.2.0",
    "ephemeral_id": "c36e9193-8e3f-4962-afce-d0c55ea77474",
    "id": "9ca9309e-7c4e-4dbc-b7af-e07269a19681"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/home/n/go/src/github.com/elastic/beats/filebeat/test.log"
    }
  },
  "env": "sb-s2",
  "input": {
    "type": "log"
  },
  "host": {
    "name": "sleipnir"
  },
  "message": "{\"plis_exception\": \"my-source: #123 my expection message. more message\"}",
  "target_index": "stu_old_files"
}

```

I cannot see any issues. Are you sure you really changed the `target`? Or is this not the event format you are using?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 16, 2022, 7:07pm UTC](https://discuss.elastic.co/t/filebeat-bug-processor-decode-json-fields-with-usage-of-target/296972/15 "2022-03-16T19:07:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.