Filebeat and Nginx module - wrong data types in ES

[JohnParson]

I'm using Nginx module in my Filebeat agent and sending data straight to ES. All data is sent correctly but there are some issues with data types. Almost all are using "text" data type. For example nginx.access.user_agent.name and nginx.access.body_sent.bytes which should use "keyword" and "long" data types according to https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-nginx.html.

I'm running Filebeat in Docker, version is 6.5.3.

This is my filebeat.yml configuration.

---

filebeat.modules:
- module: nginx
access:
enabled: true
var.paths: ["/usr/share/filebeat/logs/nginx/*access.log"]
error:
enabled: true
var.paths: ["/usr/share/filebeat/logs/nginx/*error.log"]
- module: system
syslog:
enabled: true
var.paths: ["/usr/share/filebeat/logs/system/syslog"]
var.convert_timezone: true
name: california
fields:
env: production

setup.kibana.host: myserver:5601

output.elasticsearch:
hosts: myserver:9200
indices:
- index: "filebeat-nginx-%{+yyyy.MM.dd}"
when.contains:
fileset.module: "nginx"
- index: "filebeat-system-%{+yyyy.MM.dd}"
when.contains:
fileset.module: "system"

[Christian_Dahlqvist]

I wonder if the cause may be that the standard index template that is uploaded assumes that you do not change the index names and that you therefore have had dynamic mappings applied instead. If this is the case you should be able to copy the uploaded index template and change the index pattern that determines which indices it matches. This will however only apply to newly created indices.

[ozmhsh (Win Wang)]

Can you explain a bit more about how the index template is prepared by filebeat? Based on what I read from the document, the fields.yml determines the field mapping in the template, but I found many fields.yml in the Filebeat directory, the root level, the _meta directory in each module/fileset etc. I could not find the answer from the document (maybe I missed it).

One issue I am facing (and opened another thread but got no reply) is that @timestamp field is shown as text rather than date. I am sure I did something wrong (as in another environment it is shown as date) but could not figure out how to fix it.

[JohnParson]

Thanks Christian, you were on the right track. If you create new index in filebeat.yml you also have to
add setup.template.name and setup.template.pattern. This is actually documented in the official documentation but I missed it.

So it works now with this configuration:

indices:
- index: "filebeat-%{[beat.version]}-nginx-%{+yyyy.MM.dd}"
setup.template.name: "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"
when.contains:
fileset.module: "nginx"