# Global fields not working

**URL:** <https://discuss.elastic.co/t/global-fields-not-working/62309>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 5, 2016, 7:40pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309 "2016-10-05T19:40:01Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 5, 2016, 7:40pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/1 "2016-10-05T19:40:01Z")

</div>

I am using this sample on filebeat.yml to set global fields for all prospectors but is not working, i just see on my docs local fields but not this global fields. I am using fb 5.0-beta1

```
shipper:
  fields_under_root: true
  fields:
    aws:
      instance_id: i-33458498
      region: us-east-1
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [October 5, 2016, 7:44pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/2 "2016-10-05T19:44:13Z")

</div>

Your configuration is wrong for 5.x (it's not supposed to be nested under `shipper`). See the example here [https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-general.html#libbeat-configuration-fields](https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-general.html#libbeat-configuration-fields) or in the included `/etc/filebeat/filebeat.full.yml`.

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 5, 2016, 8:51pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/3 "2016-10-05T20:51:28Z")

</div>

Thanks! not nesting under shipper worked.

```
  fields:
    aws:
      instance_id: i-33458498
      region: us-east-1

```

is not possible to use ignore\_older as global option also for all prospectors instead of repeting it 41 times in all my prospectors?

I want that first time filebeat is started to not send all the older logs, just new logs or logs since 1h ago.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [October 5, 2016, 9:02pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/4 "2016-10-05T21:02:49Z")

</div>

> [@agonzalez](#):
>
> is not possible to use ignore\_older as global option also for all prospectors instead of repeting it 41 times in all my prospectors?

No, that's not a global config option.

If you don't want to duplicate the actual value you can use a reference variable. But you still have to reference it in each prospector. [Reference Variables | Beats Platform Reference [5.0] | Elastic](https://www.elastic.co/guide/en/beats/libbeat/5.0/config-gile-format-refs.html)

```auto
fb.global_ignore_older: 1h

filebeat.prospectors:
- paths:
    - /var/log/messages
  ignore_older: ${fb.global_ignore_older}

```

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 5, 2016, 9:04pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/5 "2016-10-05T21:04:53Z")

</div>

Thanks!! is there any other way to avoid filebeat sending old logs when starting for first time or deleting registry? or is this the only property i have to set?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [October 5, 2016, 9:20pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/6 "2016-10-05T21:20:42Z")

</div>

This is the only config setting to influence that behavior. It looks at the modified time of the file. If it's within the ignore\_older period then the file is harvested from the start (which could include log lines older than that period).

If you want to filter individual log lines by timestamp then you should add Logstash into the mix. Within Logstash you can then parse the timestamps from the log line then apply a ruby filter that does timestamp math and drops older log lines.

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 6, 2016, 5:07pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/7 "2016-10-06T17:07:44Z")

</div>

Thanks, also all my logs have same multine pattern is it possible to specify something global instead of reapeating 41 times:

multiline:  
pattern: "^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}"  
negate: true  
match: after

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [October 6, 2016, 5:41pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/8 "2016-10-06T17:41:45Z")

</div>

You can use a reference variable again. For example:

```auto
fb.multiline_log4j:
  pattern: '\$\$\$'
  negate: false
  match: before

filebeat.prospectors:
  - input_type: log
    paths:
      - input.txt
    multiline: ${fb.multiline_log4j}

```

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 13, 2016, 9:49pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/9 "2016-10-13T21:49:52Z")

</div>

Thanks, that worked fine!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2016, 7:40pm UTC](https://discuss.elastic.co/t/global-fields-not-working/62309/10 "2016-10-26T19:40:15Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.
