Filebeat: filebeat.full.yml attribute duplication

Elastic Stack Beats
1

It appears in the filebeat.full.yml that their is some duplication in the below attributes.

fields:

fields_under_root:

If I set the second set fields and fields_under_root nothing happens ... to get the desired effect the first set needs to be set.

All and all a bit confusing ... took me a couple cycles to figure out i wasn't using the "second" set.

thanks in advance,

JG

2

These are not duplicates. There are different places in processing pipeline fields can be added. The first option is per prospector (one can configure multiple prospectors with potentially different fields). The second option is used by publisher pipeline ( I think ) and should add those fields to all events.

3

Thanks - Based on your above comments if I added tags using the seconds set.... I should see those tags in source system (kafka in this case) . However, I see the reverse. I have to update the tags and fields in the first set to see addition in the output.

However, maybe I am looking for the tags in the wrong place? Maybe the seconds set doesn't impact the actually output data but something else?

4

Could you provide an example of the configuration where tagging is not working as expected and an output event if possible.

The per prospectors tags should be appended to the global tags and added to each event.

The per prospector fields should be merged with the global fields and added to each event. The per prospector fields will take precedence over the global ones if there are conflicts.

5

Per your request ... config and output

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/access.log

  fields:
    role: flex
    env: production
  fields_under_root: true


name: stuff_by_jaren

tags: [
  "chance",
  "drake"
  ]

fields:
  nba: thunder

output:

{
  "@timestamp": "2016-05-29T01:54:12.753Z",
  "beat": {
    "hostname": "ip-X-X-X-X",
    "name": "ip-X-X-X-X"
  },
  "env": "production",
  "input_type": "log",
  "message": "127.0.0.1 - - [27/May/2016:18:44:07 -0400] \"GET / HTTP/1.1\" 404 19 \"-\" \"stuff here\"",
  "offset": 000000,
  "role": "flex",
  "source": "/var/log/nginx/access.log",
  "type": "log"
}
6

I just tried to reproduce this on master and couldn't. What is the exact version of filebeat you are using?

7

I am using Alpha.

I can reproduce it on the same box. However, last night I installed on a different box and it worked as expected.

Odd... I will follow up if i see it again. thx

8

Ok, thanks. My first guess is normally a config file indentation issue, but your config above looks fine.