I am trying to add fields to filebeat but cannot get it working. When my logs are indexed i cannot find the 'resource' field within my access or error logs from nginx.
I have deleted all filebeat indexes and templates from the kibana console using DELETE filebeat-* and DELETE _template/filebeat-6.8.0
This is a sample : www.my.domain.com 172.xxx.xxx.xxx - [03/Jun/2019:08:29:50 +0200] "GET /my/path/to/url/ HTTP/2.0" 200 4383 "https://my/path/to/url/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 0.067 0.064 - http_x_forwarded_for: - - proxy_add_x_forwarded_for: 172.xxx.xxx.xxx
I had to modify the ingest configuration as this is not a standard nginx grok pattern.
All the fields are ingested and parsed properly within elasticsearch.
I also see the 'resource' field within the kibana index but no document contains it.
The 'resource' field does not reside within the access log. I want to define it manually.
There are 3 NginX instances running on the same machine so I the resource field is going to be used to be able to make a distinction. This could also be accomplished by the source file location but i just want to add a custom field.
With metricbeat I just added in modules.d/nginx.yml :
Maybe instead of trying to debug this issue there is a standard way of defining extra fields as in the case of metricbeat?
Is there any documentation regarding this?
I know that extra fields can be set using the inputs section in filebeat.yml but this is not the specific use case.
My question thus becomes :
How do i add custom fields from within the modules.d/nginx.yml file?
Your configuration is incorrect, that's why you are not seeing the field. fields and other input level settings go inder the keyword input in case of Filebeat.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.