Metricbeat Dynamic Top Level Fields

[icozamiz (isaabelaa_07@hotmail.com)]

Hello!

I am new to beats and am currently working on a module in metricbeat that polls a system through a webservice call and gets a usage summary of the system and it's subsystems in the form a nested json object. In my go files I am able to split up the large event into smaller events (by subsystem).

I want to be able to include the subsystem field as a top level field (alongside the metricset name). I know that top level fields can be implemented statically in the metricbeat.yml config file but I was wondering if there a way to do this dynamically? I thought about creating a metricset for each subsystem but I prefer to do one fetch and then split the response into multiple events. Another solution I thought of would be to populate other metricsets from a main metricset that performs the single fetch (per period).

P.S beats are awesome

[ruflin]

We require the structure module.metricset to make sure the experience is the same across all metricset implementation. So an implementation of a Metricset cannot push fields to the top level. This also prevents potential conflicts.

To create multiple documents for one Fetch, have a look at the MultiFetch interface: https://www.elastic.co/guide/en/beats/metricbeat/current/creating-metricsets.html#_multi_fetching The only difference is, that it returns a list of ´[]common.MapStr´ instead of just a ´common.MapStr´

If the documents have different structures, different metricsets should be used as otherwise type conflicts could occur in elasticsearch.

Perhaps you can share some more details about the metricset you are building and the overall structure of the documents? Make it easier to help

[icozamiz (isaabelaa_07@hotmail.com)]

I am currently using the MultiFetch interface so that my Fetch command creates multiple different events each differentiated by the "subsystem" field. In the future I want to make different modules for different systems that follows this same idea. Each module queries the system through an http web service call and received a response that states what subsystems are currently running within the system.

Ideally each module should only have one metricset that performs one fetch and then creates multiple events with a structure of moduleName.subsystem instead of the current moduleName.status.subsystem structure. My current metricset is called status and it create a list of []common.MapStr events that each have an outer field called subsystem and inner fields that contain the details of the subsystem instance.