Filebeat: Bug: Processor "decode_json_fields" with usage of "target"

Elastic Stack Beats
1

OS: SLES12, CentOS 7.9
Filebeat Versions: 7.8.0, 7.16.2

After using processor "decode_json_fields" WITH "target: 'sometarget' it's impossible to access some extracted json fields with following processors.

If you let "target": '' (unset) the following processors are able to access the extracted json fields.

##############################
#  running fine
  processors:
    - if:
        regexp:
          message: '"parameter\d+":\w+\.?\d*'
      then:
        - add_fields:
            target: ''
            fields:
              not_isy: Parameter_not_string

    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: ""
        overwrite_keys: false
        add_error_key: true
...
# This can extract from field "exception"
    - dissect:
        tokenizer: '%{exception_source}: #%{exception_code} %{exception_err_msg}. %{exception_rest}'
        field: exception
        target_prefix: ''

############################
# NOT running fine
  processors:
    - if:
        regexp:
          message: '"parameter\d+":\w+\.?\d*'
      then:
        - add_fields:
            target: ''
            fields:
              not_isy: Parameter_not_string

    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: "mytarget"
        overwrite_keys: false
        add_error_key: true
...
# Never doing anything
    - dissect:
        tokenizer: '%{exception_source}: #%{exception_code} %{exception_err_msg}. %{exception_rest}'
        field: mytarget_exception
        target_prefix: 'mytarget'

I have made every attempt I can imagine: e.g. "if has_fields" cannot detect the field, but it's definitely present. Same with regex ...
I tried e.g.:

#    - if:
#        has_fields: ['mytarget_exception']
#      then:
#        - add_fields:
#            target: 'mytarget'
#            fields:
#              exception_found: yes

I also tried with "has_fields: ['exception']" ... no success

Every time I worked with usage of "target" or "target_prefix" it was failing ...
Now I work without that and my 322-line-filebeat-config does (nearly) what I want.

There are some more issues like "multiline pattern only working with type "log" but NOT with type "filestream" and so on. But that in another issue ... :wink:

2

To me, it seems like misconfiguration. You put the results of the JSON parsing to under the field mytarget, but in the dissect processor you try to parse the field mytarget_exception which is I assume is empty.

But it would be nice to see an example event you are trying to parse.

There are some more issues like "multiline pattern only working with type "log" but NOT with type "filestream" and so on. But that in another issue ... :wink:

Then please post an another question... :wink:

3

Hi,
"decode_json_fields" extracts the field "exception" which I want to deal with other processors.
This works only WITHOUT setting "target" to something else than empty (''). Please see my 1st code-example with "# running fine" comment.

Unfortunately I must not give a real life example.

4

If you set the target of decode_json_fields to an empty value, Filebeat puts the fields to the root of the event. I assume one of the parsed fields is called exception. Then in the later dissect processor, you configure it as the source, and it can be parsed as expected.

However, in your second configuration snippet that does not work you put the results of the JSON processor under mytarget. Your expcetion field lands under the name mytarget.exception. So you should configure mytarget.exception as the field of dissect processor. I assume that is the field you would like to dissect.

5

I tried that, see 2nd example with comment "# never doing anything"

6

No, in your example you have mytarget_exception, not mytarget.exception.

7

I also tried that. "mytarget.exception" was my last try ... sorry for beeing not so clear!

8

I tried and analysed for appr. 8h, with filebeat debug logs, looking into Graylog for which fields where extracted, processed and so on.
My "running" filebeat config (without any "target"-setting) has 322 lines, including multiline pattern in other inputs. So I assume to have understood most of the concept.

9

Let's take a step back. What if you temporarily remove the dissect processor and write to file output to see where the fields from decode_json_fields land?

10

Scenario 1: target: "mytarget":
If I look into Graylog, I can see the field "mytarget_exception", also all other json-decoded fields. But in filebeat I cannot access, no processor can "see" this field(s) : no "IF hasfield", no "dissect" ,...

Scenario 2: target unset:
All expected json_decoded fields AND processed derivates of them are present in Graylog. So in Filebeat I can process json_decoded fields with "dissect", I can process with "IF hasfield ..." ...

11

Are there any other processors or parsers besides the ones you shared? I am trying to track down where exactly the field is born and why it is not "visible" to the later processor.

12

Here is the complete section which is NOT running related to usage of "target" or "target_prefix".

If you don't use "target" (and change appropriate fields) everything is fine (fields just have no prefix in Graylog)

#=========================== Filebeat inputs =============================

##### global options #####
processors:
  - add_host_metadata: ~

filebeat.inputs:
#####  
- type: log
  enabled: true
  fields_under_root: true
  ignore_older: 48h
  max_bytes: 32000
  scan_frequency: 10s

  paths:
    - /var/log/....

  fields:
    input_tag: plis
    target_index: stu_old_files
    env: sb-s2


  # not_isy correction
  processors:
    - if:
        regexp:
          message: '"parameter\d+":\w+\.?\d*'
      then:
        - add_fields:
            target: ''
            fields:
              not_isy: Parameter_not_string

    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: "plis"
        overwrite_keys: false
        add_error_key: true

    # Alle Felder: string-Format sicherstellen
    - convert:
        fields:
          - {from: "plis_parameter1", type: "string"}
          - {from: "plis_parameter2", type: "string"}
          - {from: "plis_parameter3", type: "string"}
          - {from: "plis_parameter4", type: "string"}
#    test      - {from: "parameter4", to: "parameter4_convert", type: "string"}
          - {from: "plis_parameter5", type: "string"}
          - {from: "plis_parameter6", type: "string"}
          - {from: "plis_parameter7", type: "string"}
          - {from: "plis_parameter8", type: "string"}
          - {from: "plis_parameter9", type: "string"}
          - {from: "plis_parameter10", type: "string"}
          - {from: "plis_parameter11", type: "string"}
          - {from: "plis_parameter12", type: "string"}
        ignore_missing: true
        fail_on_error: false
        mode: copy

#    # application aus log_file_path
    - dissect:
        tokenizer: '/var/log/%{application}/%{log_file_name}'
        # not running: field: log_file_path
        field: log.file.path
        target_prefix: 'plis'


    # Exception code
    - if:
        regexp:
          plis_exception: ": #"
      then:
        - dissect:
##            tokenizer: '%{plis_exception_source}: #%{plis_exception_code} %{plis_exception_err_msg}. %{plis_exception_prelude} Caused by: %{plis_exception_cause} at %{plis_exception_rest}'
            tokenizer: '%{exception_source}: #%{exception_code} %{exception_err_msg}. %{exception_rest}'
            field: plis_exception
            target_prefix: 'plis'
      else:
        - dissect:
            tokenizer: '%{exception_err_msg}: %{exception_rest}'
            field: plis_exception
            target_prefix: 'plis'

    # ggf. erzeugtes extract-Field wieder loeschen
    - drop_fields:
        fields: ["plis_exception_rest"]
13

Hello Noémi, anything new on this issue?

14

I tried to came up with an example event based on your configuration. I used this:

{"plis_exception": "my-source: #123 my expection message. more message"}

I changed the targets to plis.plis_expection in the exception handling dissect processors and got this event:

{
  "@timestamp": "2022-02-16T17:03:13.272Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.2.0"
  },
  "input_tag": "plis",
  "plis": {
    "exception_err_msg": "my expection message",
    "plis_exception": "my-source: #123 my expection message. more message",
    "application": "filebeat",
    "log_file_name": "test.log",
    "exception_rest": "more message",
    "exception_source": "my-source",
    "exception_code": "123"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "name": "sleipnir",
    "type": "filebeat",
    "version": "8.2.0",
    "ephemeral_id": "c36e9193-8e3f-4962-afce-d0c55ea77474",
    "id": "9ca9309e-7c4e-4dbc-b7af-e07269a19681"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/home/n/go/src/github.com/elastic/beats/filebeat/test.log"
    }
  },
  "env": "sb-s2",
  "input": {
    "type": "log"
  },
  "host": {
    "name": "sleipnir"
  },
  "message": "{\"plis_exception\": \"my-source: #123 my expection message. more message\"}",
  "target_index": "stu_old_files"
}

I cannot see any issues. Are you sure you really changed the target? Or is this not the event format you are using?

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.