# Help needed for setup.template.append\_fields usage

**URL:** <https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 31, 2019, 10:09am UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701 "2019-05-31T10:09:39Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Georgios\_Gkinis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgios_gkinis/32/47225_2.png) [@Georgios\_Gkinis](https://discuss.elastic.co/u/Georgios_Gkinis)\
**Post date:** [May 31, 2019, 10:09am UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701/1 "2019-05-31T10:09:39Z")

</div>

I am trying to add fields to filebeat but cannot get it working. When my logs are indexed i cannot find the 'resource' field within my access or error logs from nginx.

I have deleted all filebeat indexes and templates from the kibana console using  
**DELETE filebeat-** \* and **DELETE \_template/filebeat-6.8.0**

Can someone point out what I am doing wrong?

filebeat.yml:

```auto
#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

setup.template.overwrite: true
setup.template.append_fields:
  - name: nginx.resource
    type: keyword

```

/modules.d/nginx.yml :

```auto
- module: nginx
  resource: 'azn'
  # Access logs
  access:
   enabled: true
   var.paths: ["/path/to/azn/root_access.log"]

  # Error logs
  error:
   enabled: true
   var.paths: ["/path/to/azn/error.log"]

```

---

<div class="post-metadata">

**Author:** ![faec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faec/32/46988_2.png) [@faec](https://discuss.elastic.co/u/faec)\
**Post date:** [May 31, 2019, 8:11pm UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701/2 "2019-05-31T20:11:08Z")

</div>

Hi! Could you give an example log line where you're seeing the problem, and how it looks after being indexed?

---

<div class="post-metadata">

**Author:** ![Georgios\_Gkinis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgios_gkinis/32/47225_2.png) [@Georgios\_Gkinis](https://discuss.elastic.co/u/Georgios_Gkinis)\
**Post date:** [June 3, 2019, 9:57am UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701/3 "2019-06-03T09:57:25Z")

</div>

Dear Fae,

This is a sample :  
**[www.my.domain.com](http://www.my.domain.com) 172.xxx.xxx.xxx - [03/Jun/2019:08:29:50 +0200] "GET /my/path/to/url/ HTTP/2.0" 200 4383 "[https://my/path/to/url/](https://my/path/to/url/)" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 0.067 0.064 - http\_x\_forwarded\_for: - - proxy\_add\_x\_forwarded\_for: 172.xxx.xxx.xxx**

I had to modify the ingest configuration as this is not a standard nginx grok pattern.  
All the fields are ingested and parsed properly within elasticsearch.  
I also see the 'resource' field within the kibana index but no document contains it.

The **'resource'** field does not reside within the access log. I want to define it manually.  
There are 3 NginX instances running on the same machine so I the resource field is going to be used to be able to make a distinction. This could also be accomplished by the source file location but i just want to add a custom field.

With metricbeat I just added in modules.d/nginx.yml :

```auto
- module: nginx
  metricsets:
    - stubstatus
  period: 10s

  fields:
    resource: 'diensten'

```

and it works like a charm!

I am trying to do the same with filebeat, that's all.

Thank you for your time.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [June 3, 2019, 10:07am UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701/4 "2019-06-03T10:07:21Z")

</div>

I cannot see the `fields` setting in your Filebeat configuration. Where are you setting it?

---

<div class="post-metadata">

**Author:** ![Georgios\_Gkinis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgios_gkinis/32/47225_2.png) [@Georgios\_Gkinis](https://discuss.elastic.co/u/Georgios_Gkinis)\
**Post date:** [June 3, 2019, 12:00pm UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701/5 "2019-06-03T12:00:53Z")

</div>

Even if i set it it has no effect :

/modules.d/nginx.yml

```auto
- module: nginx

  # Access logs
  access:
    fields:
      resource: 'idp'
    enabled: true
    var.paths: ["/tmp/root_access.log"]

  # Error logs
  error:
    fields:
      resource: 'idp'
    enabled: true
    var.paths: ["/tmp/error.log"]

```

As mentioned i can find the field in the kibana index, so template uploading works well.  
The issue is that the field is not being populated.

---

<div class="post-metadata">

**Author:** ![Georgios\_Gkinis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgios_gkinis/32/47225_2.png) [@Georgios\_Gkinis](https://discuss.elastic.co/u/Georgios_Gkinis)\
**Post date:** [June 3, 2019, 12:49pm UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701/6 "2019-06-03T12:49:03Z")

</div>

Maybe instead of trying to debug this issue there is a standard way of defining extra fields as in the case of metricbeat?  
Is there any documentation regarding this?  
I know that extra fields can be set using the inputs section in filebeat.yml but this is not the specific use case.

My question thus becomes :

- How do i add custom fields from within the modules.d/nginx.yml file?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [June 3, 2019, 1:54pm UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701/7 "2019-06-03T13:54:37Z")

</div>

Your configuration is incorrect, that's why you are not seeing the field. `fields` and other `input` level settings go inder the keyword `input` in case of Filebeat.

```auto
- module: nginx

  # Access logs
  access:
    enabled: true
    input:
      fields:
        resource: 'idp'
    var.paths: ["/tmp/root_access.log"]

  # Error logs
  error:
    input:
      fields:
        resource: 'idp'
    enabled: true
    var.paths: ["/tmp/error.log"]

```

---

<div class="post-metadata">

**Author:** ![Georgios\_Gkinis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgios_gkinis/32/47225_2.png) [@Georgios\_Gkinis](https://discuss.elastic.co/u/Georgios_Gkinis)\
**Post date:** [June 3, 2019, 2:02pm UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701/8 "2019-06-03T14:02:08Z")

</div>

I found the solution :

**modules.d/nginx.yml :**

```auto
- module: nginx
  # Access logs
  access:
    input:
      fields:
        resource: 'idp'
    enabled: true
    var.paths: ["/tmp/root_access.log"]

```

There is no need to define :  
**filebeat.yml**

```auto
setup.template.overwrite: true
setup.template.append_fields:
  - name: resource
    type: keyword

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 1, 2019, 2:02pm UTC](https://discuss.elastic.co/t/help-needed-for-setup-template-append-fields-usage/183701/9 "2019-07-01T14:02:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.