Allow for renaming of json fields in http module output

(Stefan Pfeiffer) #1

I want to ingest Spring Boot Actuator metrics information (/metrics endpoint). The http module takes the JSON returned by my spring boot application and feeds it to ES using the elasticsearch output.

Basically, i run into this problem:

There are JSON fields named classes and classes.loaded with an integer value. ES helpfully converts the dotted notation of classes.loaded to {"classes":{"loaded":12345}}, a nested JSON object. Unfortunately, classes: 12345 cannot be parsed/indexed as classes already is an object which cannot contain a long primitive.

Obviously. there are three workarounds right now:

  1. Feed the data to logstash and use the de_dot filter. This is computational intensive and needs a logstash server just for this use case. I would like to omit logstash as long as possible and go the route of ingest pipelines. Unfortunately, ingest pipelines do not offer anything resembling the de_dot filter.
  2. Drop the fields that lead to problems, e.g. classes in the example given. But i have lots of those conflicting fields, and i am not fond of dropping them all. Especially as i lose the data these fields contain. Different Spring Boot services add different metrics values dynamically, calling for perpetual adaption of the list of fields to drop to get rid of the offending fields.
  3. Using the community spring beat
    Unfortunately, there is no binary release available, and i am not really into building this stuff myself. As i am using metricbeat anyway for system metrics, i also wanted to avoid using more independent beats for the different metrics, i would prefer a module.

The jolokia module seems to suffer from the same problem:

The httpbeat from which the http module in metricbeat inherited already solved that problem. It had a configurable replacement of dots in the JSON field names:

Unfortunately, the http module has no such capability.

Given the abundance of Spring Boot applications and the need for monitoring them using actuator endpoints, i am surprised there is not so much information on how to tackle the problem efficiently. Are there plans to incorporate a de_dot like capability in the http module of metricbeat, like httpbeat already had it?


(Stefan Pfeiffer) #2

The problem also surfaced on the Spring Boot Actuator side of things:

Obviously, the problem will vanish in Spring Boot 2, but i am still bound to 1.5 unfortunately :frowning:

(ruflin) #3

@spfeiffer Thanks for bringing this up. I agree this is a problem we need to address as it affects multiple parts in Beats. Can you open a Github issue an we can discuss there on what the best place to implement is.

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.