Dynamic Root Field

Elastic Stack Beats
1

How could I modify a beat to apply a dynamic root field to all events?

For instance, I want to modify metricbeat and apply a set of tags for changing ongoing events to every metric that is collected so I can correlate on metrics on this data.

2

Hi @jcatana,

That's an interesting use case, we have done things like this in the past, normally through processors. Could you explain a bit on what you are seeking? It may make sense to offer this feature in a generic way so everyone can benefit from it :slight_smile:

3

I run a large HPC center. I would like to correlate system metrics to jobs. Jobs are transient coming and going from multiple systems during the day.
I want to easily correlate the metrics recorded to the job that is running or ran.
Being able to globally tag each event recorded with the job ID or IDs currently running seems like the easiest option to make use of the filter functionality provided in the kibana dashboards.

I've been able to write my own module to gather the data and create events, but I can't figure out how to hook in to make it globally tag all events the are recorded.

4

Would the fields setting work for you in this case? Have a look at https://www.elastic.co/guide/en/beats/metricbeat/5.6/configuring-howto-metricbeat.html for details

5

A field may work, but the documentation in that link is not very informative.

  1. I need this to be updated dynamically. Preferably without restarting the service or reloading the config file.

  2. I may have multiple jobs on the same system which is why tags were appealing.
    I would need something like
    fields:
    jobid: ["10.job", "11.job", "12.job"]

or
fields:
jobid: "10.job"
jobid: "11.job"
jobid: "12.job"

I'm guessing we cannot have duplicate keys.

6

I'm wondering, why don't you add this info to your module events?

7

I want the field/tag to be in the events for all modules of the beat, not only my module. How would I do that?

8

Then your best shot is to write a new processor, check https://github.com/elastic/beats/tree/master/libbeat/processors/add_locale as a simple example :slight_smile:

9

This appears to be exactly what I'm looking for. I'll look into it and let you know how it turns out.

10

Yep, it works perfectly. Thanks a ton for helping me locate this and figure this out!

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.