Filebeat and Nginx module - wrong data types in ES

I'm using Nginx module in my Filebeat agent and sending data straight to ES. All data is sent correctly but there are some issues with data types. Almost all are using "text" data type. For example and nginx.access.body_sent.bytes which should use "keyword" and "long" data types according to

I'm running Filebeat in Docker, version is 6.5.3.

This is my filebeat.yml configuration.

- module: nginx
enabled: true
var.paths: ["/usr/share/filebeat/logs/nginx/*access.log"]
enabled: true
var.paths: ["/usr/share/filebeat/logs/nginx/*error.log"]
- module: system
enabled: true
var.paths: ["/usr/share/filebeat/logs/system/syslog"]
var.convert_timezone: true
name: california
env: production myserver:5601

hosts: myserver:9200
- index: "filebeat-nginx-%{+yyyy.MM.dd}"
fileset.module: "nginx"
- index: "filebeat-system-%{+yyyy.MM.dd}"
fileset.module: "system"

I wonder if the cause may be that the standard index template that is uploaded assumes that you do not change the index names and that you therefore have had dynamic mappings applied instead. If this is the case you should be able to copy the uploaded index template and change the index pattern that determines which indices it matches. This will however only apply to newly created indices.

Can you explain a bit more about how the index template is prepared by filebeat? Based on what I read from the document, the fields.yml determines the field mapping in the template, but I found many fields.yml in the Filebeat directory, the root level, the _meta directory in each module/fileset etc. I could not find the answer from the document (maybe I missed it).

One issue I am facing (and opened another thread but got no reply) is that @timestamp field is shown as text rather than date. I am sure I did something wrong (as in another environment it is shown as date) but could not figure out how to fix it.

Thanks Christian, you were on the right track. If you create new index in filebeat.yml you also have to
add and setup.template.pattern. This is actually documented in the official documentation but I missed it.

So it works now with this configuration:

- index: "filebeat-%{[beat.version]}-nginx-%{+yyyy.MM.dd}" "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"
fileset.module: "nginx"

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.