Using a single metricbeat agent to finetune settings before a larger deployment to 3000+ hosts, and trying to contain the number of fields to only those required, I seem to hit a wall at the "metricsets" level. I would like to configure more finely the exact fields I require (or remove those I don't).
For example :
In the metricbeat/modules.d/system.yml
I can activate/deactivate the - load metricset.
However, how could I choose to only send load1 (or exclude load5 and load15) ?
Alternatively, maybe, and if all else fails, which is the best way to look for filtering out fields at the ingest stage from the receiving side of the elasticsearch instance... There is no logstash on elaticcloud. Would I be looking at ingest pipelines ?
Right. So I received confirmation from support that we cannot select a subset of fields for any given metricset of a beat agent module. That closes that part definitely (only alternative for not shipping unwanted fields would be to create your own agent).
I wonder if this is also true when using the fleet management feature (I suppose yes...).
Anyhow, looking into ingest node pipelines processors seems indeed to be the cheapest route to filtering out unwanted fields before index insertion.
Now, still reading up on processor possibilities, I only seem so far to have found rules to exclude certain fields (with use of the remove processor ).
Again, if I could just select the wanted fields, instead of excluding all unwanted could be a time saver (and potentially more ressource efficient).... To be continued...
You might want to look at the drop_fields processor of Metricbeat. It does effectively the same thing as the remove processor, but on the agent side, thus reducing what you need to send.
Wow. Really ? Before they are shipped ?
Thank you !
Do you configure this in each module's setting module.yml, or in the main metricbeat.yml ?
[edit] [self-answer] The documentation states both places are possible...
I've selected my recap above as a solution for the thread because it groups all the findings on the topic, but you've actually nailed it for my question. Aligatô !
Funny how the elasticsearch support team told me (and in brief words) how this was impossible...
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.