Filebeat add_fields processor

Elastic Stack Beats
1

Hello colleagues;

I am trying to add an ECS event.dataset with the add_fields processor similar to several of the Filebeat modules e.g., the Apache module which add the event datasets apache.access and apache.error.

I use the add_fields processor and configure it with either

target: ''

or

events_under_root: true

However what I get is the event.data set under the fields key:

fields:
  event.dataset: "app.access"

What I expect is the event.dataset at the root level of the document

event.dataset: "app.access"

e.g. as all the Filebeat modules do. What am I missing?

I know I can use the add_tags processor but I am already using this for other purposes.

BR
Flemming

2

Hi @fgjensen

Did you try exactly like this? I just did and it worked as expected.

processors:
  - add_fields:
      target: ''
      fields:
        event.dataset: "app.access"

The events_under_root is not a valid setting in the add_fields processor.

There could be a subtle conflict ...are you using a module or just a generic input? If you are using a module like ngnix or something it could be overwriting event.dataset... I would need to test that case.

3

Hi @stephenb

Thanks for taking time to addressing this question.

I configured the processors exactly as you suggests. It did not work for me.

I should add that the inputs are filestreams, several filestream inputs are configured in one configuration file under the input.d directory and each input has more than one processor defined.

I can share the configuration file if it helps you.

BR
Flemming

4

sure if you like I suspect the processors are not in the right place, you should probably try putting them aligned with each input... see here

Its hard to tell when I can only see snippets

1 Like
5

Hi @stephenb

I reviewed my implementation and must admit I had an error.

This works for the add_fields processor and the dissect processor in a filestream input.

    processors:
      - add_fields:
          target: ''
          fields:
            event.dataset: "cas.log"
      - if:
          regexp:
            message: '\s\bWARN\s|\s\bERROR\s' # Fetch WARN or ERROR levels into the log.level field
        then:
          - dissect:
              tokenizer: "%{} %{log.level} %{}"
              field: "message"
              target_prefix: ''
              ignore_failure: true
              ignore_missing: true
              overwrite_keys: true

Thanks for your help.

Best regards
Flemming

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.