Global fields not working

Elastic Stack Beats
1

I am using this sample on filebeat.yml to set global fields for all prospectors but is not working, i just see on my docs local fields but not this global fields. I am using fb 5.0-beta1

shipper:
  fields_under_root: true
  fields:
    aws:
      instance_id: i-33458498
      region: us-east-1
2

Your configuration is wrong for 5.x (it's not supposed to be nested under shipper). See the example here https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-general.html#libbeat-configuration-fields or in the included /etc/filebeat/filebeat.full.yml.

3

Thanks! not nesting under shipper worked.

  fields:
    aws:
      instance_id: i-33458498
      region: us-east-1

is not possible to use ignore_older as global option also for all prospectors instead of repeting it 41 times in all my prospectors?

I want that first time filebeat is started to not send all the older logs, just new logs or logs since 1h ago.

4

No, that's not a global config option.

If you don't want to duplicate the actual value you can use a reference variable. But you still have to reference it in each prospector. Reference Variables | Beats Platform Reference [5.0] | Elastic

fb.global_ignore_older: 1h

filebeat.prospectors:
- paths:
    - /var/log/messages
  ignore_older: ${fb.global_ignore_older}
5

Thanks!! is there any other way to avoid filebeat sending old logs when starting for first time or deleting registry? or is this the only property i have to set?

6

This is the only config setting to influence that behavior. It looks at the modified time of the file. If it's within the ignore_older period then the file is harvested from the start (which could include log lines older than that period).

If you want to filter individual log lines by timestamp then you should add Logstash into the mix. Within Logstash you can then parse the timestamps from the log line then apply a ruby filter that does timestamp math and drops older log lines.

7

Thanks, also all my logs have same multine pattern is it possible to specify something global instead of reapeating 41 times:

multiline:
pattern: "^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}"
negate: true
match: after

8

You can use a reference variable again. For example:

fb.multiline_log4j:
  pattern: '\$\$\$'
  negate:  false
  match:   before

filebeat.prospectors:
  - input_type: log
    paths:
      - input.txt
    multiline: ${fb.multiline_log4j}
9

Thanks, that worked fine!

10

This topic was automatically closed after 21 days. New replies are no longer allowed.