Filebeat Affecting Elasticsearch Dynamic Mapping

[cb2015 (Cody Burke)]

Hi Everyone,

I am using filebeat to send logs to logstash for processing, and finally, sending them to elasticsearch for graphing in kibana.

My question is:

Does filebeat affect how elasticsearch makes a dynamic mapping?

I ask this because when I used the "file" input to send logs to elasticsearch, I got very simple dynamic mappings(a dozen lines). However when I changed the logstash input to be filebeat, The mapping became huge (hundreds of lines). In addition, some fields now make Kibana unhappy, as I get an error I did not get before.

Does anyone know if filebeat has an effect on elasticsearch dynamic mappings, and if so, how do I control it?

Any help would be greatly appreciated.

[steffens (Steffen Siering)]

Which filebeat version? have you registered a mapping template with elasticsearch? Filebeat 5.0 alpha3 will install the mapping template if no mapping template does exist. Logstash can install some template too, if it does not exist (depends on your config). If no mapping (template) is available in elasticsearch, elasticsearch will automatically generate the mapping depending on events being received. In kibana you can try to 'reload' the mapping info in 'Settings'.

[ruflin]

Filebeat provides its own template. It is automatically loaded in the newer version of filebeat, but you can also turn it off: https://www.elastic.co/guide/en/beats/filebeat/1.2/elasticsearch-output.html#_template

As the filebeat template is quite simple I'm somehow surprised about the (hundreds of lines).