Defining fields for metricsets

[efftee]

Hi there,

Using a single metricbeat agent to finetune settings before a larger deployment to 3000+ hosts, and trying to contain the number of fields to only those required, I seem to hit a wall at the "metricsets" level. I would like to configure more finely the exact fields I require (or remove those I don't).

For example :
In the metricbeat/modules.d/system.yml
I can activate/deactivate the - load metricset.
However, how could I choose to only send load1 (or exclude load5 and load15) ?

Best regards,

[efftee]

Alternatively, maybe, and if all else fails, which is the best way to look for filtering out fields at the ingest stage from the receiving side of the elasticsearch instance... There is no logstash on elaticcloud. Would I be looking at ingest pipelines ?

[warkolm (Mark Walkom)]

I've moved this to the Beats category to see if anyone can help there.

@efftee

There is no logstash on elaticcloud. Would I be looking at ingest pipelines ?

Definitely.

[efftee]

Right. So I received confirmation from support that we cannot select a subset of fields for any given metricset of a beat agent module. That closes that part definitely (only alternative for not shipping unwanted fields would be to create your own agent).

I wonder if this is also true when using the fleet management feature (I suppose yes...).

Anyhow, looking into ingest node pipelines processors seems indeed to be the cheapest route to filtering out unwanted fields before index insertion.

Now, still reading up on processor possibilities, I only seem so far to have found rules to exclude certain fields (with use of the remove processor ).
Again, if I could just select the wanted fields, instead of excluding all unwanted could be a time saver (and potentially more ressource efficient).... To be continued...

[efftee]

So just as a conclusion to this question for any forum dwellers who might tumble here :

• You may filter out fields directly from the agent's main configuration, or from specific modules configurations, by using the drop-field processor ;

• You may alternatively filter them out through a dedicated ingest node pipeline by using a remove processor.

[BenB196]

You might want to look at the drop_fields processor of Metricbeat. It does effectively the same thing as the remove processor, but on the agent side, thus reducing what you need to send.

[efftee]

Wow. Really ? Before they are shipped ?
Thank you !

Do you configure this in each module's setting module.yml, or in the main metricbeat.yml ?
[edit] [self-answer] The documentation states both places are possible...

Thanks !

[BenB196]

You can do either/or. This part of the docs explains it: https://www.elastic.co/guide/en/beats/metricbeat/current/defining-processors.html#where-valid

[efftee]

Thank you ever so much. You've been spot on !

I've selected my recap above as a solution for the thread because it groups all the findings on the topic, but you've actually nailed it for my question. Aligatô !

Funny how the elasticsearch support team told me (and in brief words) how this was impossible...