I am using filebeat to send logs to logstash for processing, and finally, sending them to elasticsearch for graphing in kibana.
My question is:
Does filebeat affect how elasticsearch makes a dynamic mapping?
I ask this because when I used the "file" input to send logs to elasticsearch, I got very simple dynamic mappings(a dozen lines). However when I changed the logstash input to be filebeat, The mapping became huge (hundreds of lines). In addition, some fields now make Kibana unhappy, as I get an error I did not get before.
Does anyone know if filebeat has an effect on elasticsearch dynamic mappings, and if so, how do I control it?
Which filebeat version? have you registered a mapping template with elasticsearch? Filebeat 5.0 alpha3 will install the mapping template if no mapping template does exist. Logstash can install some template too, if it does not exist (depends on your config). If no mapping (template) is available in elasticsearch, elasticsearch will automatically generate the mapping depending on events being received. In kibana you can try to 'reload' the mapping info in 'Settings'.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.