Help needed for setup.template.append_fields usage

Elastic Stack Beats
1

I am trying to add fields to filebeat but cannot get it working. When my logs are indexed i cannot find the 'resource' field within my access or error logs from nginx.

I have deleted all filebeat indexes and templates from the kibana console using
DELETE filebeat-* and DELETE _template/filebeat-6.8.0

Can someone point out what I am doing wrong?

filebeat.yml:

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

setup.template.overwrite: true
setup.template.append_fields:
  - name: nginx.resource
    type: keyword

/modules.d/nginx.yml :

- module: nginx
  resource: 'azn'
  # Access logs
  access:
   enabled: true
   var.paths: ["/path/to/azn/root_access.log"]

  # Error logs
  error:
   enabled: true
   var.paths: ["/path/to/azn/error.log"]
2

Hi! Could you give an example log line where you're seeing the problem, and how it looks after being indexed?

3

Dear Fae,

This is a sample :
www.my.domain.com 172.xxx.xxx.xxx - [03/Jun/2019:08:29:50 +0200] "GET /my/path/to/url/ HTTP/2.0" 200 4383 "https://my/path/to/url/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 0.067 0.064 - http_x_forwarded_for: - - proxy_add_x_forwarded_for: 172.xxx.xxx.xxx

I had to modify the ingest configuration as this is not a standard nginx grok pattern.
All the fields are ingested and parsed properly within elasticsearch.
I also see the 'resource' field within the kibana index but no document contains it.

The 'resource' field does not reside within the access log. I want to define it manually.
There are 3 NginX instances running on the same machine so I the resource field is going to be used to be able to make a distinction. This could also be accomplished by the source file location but i just want to add a custom field.

With metricbeat I just added in modules.d/nginx.yml :

- module: nginx
  metricsets:
    - stubstatus
  period: 10s

  fields:
    resource: 'diensten'

and it works like a charm!

I am trying to do the same with filebeat, that's all.

Thank you for your time.

4

I cannot see the fields setting in your Filebeat configuration. Where are you setting it?

5

Even if i set it it has no effect :

/modules.d/nginx.yml

- module: nginx

  # Access logs
  access:
    fields:
      resource: 'idp'
    enabled: true
    var.paths: ["/tmp/root_access.log"]

  # Error logs
  error:
    fields:
      resource: 'idp'
    enabled: true
    var.paths: ["/tmp/error.log"]

As mentioned i can find the field in the kibana index, so template uploading works well.
The issue is that the field is not being populated.

6

Maybe instead of trying to debug this issue there is a standard way of defining extra fields as in the case of metricbeat?
Is there any documentation regarding this?
I know that extra fields can be set using the inputs section in filebeat.yml but this is not the specific use case.

My question thus becomes :

  • How do i add custom fields from within the modules.d/nginx.yml file?
7

Your configuration is incorrect, that's why you are not seeing the field. fields and other input level settings go inder the keyword input in case of Filebeat.

- module: nginx

  # Access logs
  access:
    enabled: true
    input:
      fields:
        resource: 'idp'
    var.paths: ["/tmp/root_access.log"]

  # Error logs
  error:
    input:
      fields:
        resource: 'idp'
    enabled: true
    var.paths: ["/tmp/error.log"]
8

I found the solution :

modules.d/nginx.yml :

- module: nginx
  # Access logs
  access:
    input:
      fields:
        resource: 'idp'
    enabled: true
    var.paths: ["/tmp/root_access.log"]

There is no need to define :
filebeat.yml

setup.template.overwrite: true
setup.template.append_fields:
  - name: resource
    type: keyword
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.