Metricbeat pushing the whole template and the induced sparsity

[wokmichel]

Hello,
a question about metricbeat template and sparsity :

My understanding is that when setting up the metricbeat template in ElasticSearch via the following command :

metricbeat setup --template

The whole template is loaded into ElasticSearch according to what is described into fields.yml, regardless of what modules are enabled. That is to say, for instance, that, if I only want to use the system module, every document in elasticsearch will have a total of approximately 2500 fields with a vas majority of those being empty.

Although I am aware that sparsity handling has been greatly optimized and improved since the introduction of Lucene 7, I was wondering if there was something I was missing when it comes to setting up the template only for the enabled modules or if this was meant to be used this way, with the whole template loaded.

In a nutshell :
Is it a problem (considering the incuced sparsity) to have the whole, huge metricbeat template loaded in ElasticSearch or is it normal behavior ?
If not, is there a way (other than manually editing the fields.yml) to choose which part of the template to push to ElasticSearch, according to the enabled modules ?

Thanks a lot

[Kaiyan_Sheng]

Hi @wokmichel great question I usually don't even run metricbeat setup --template command since Metricbeat automatically loads the template file fields.yml anyway. I guess one thing I can think of is to modify the fields.yml to only keep the fields that you care about so you don't load all the 2500 fields.

[wokmichel]

Hello there and thanks for your answer.
As I understand from your answer, this is normal that the template is so huge and that it induces sparsity ?

Thanks

[Kaiyan_Sheng]

Yeah my understanding is, metricbeat setup --template loads everything from fields.yml. And this fields.yml is a combination of all fields.yml from all modules in metricbeat. So if want a smaller fields.yml, you can remove fields.yml under specific modules and then run make update to rebuild metricbeat.