Allow for renaming of json fields in http module output

spfeiffer · December 19, 2017, 3:25pm

I want to ingest Spring Boot Actuator metrics information (/metrics endpoint). The http module takes the JSON returned by my spring boot application and feeds it to ES using the elasticsearch output.

Basically, i run into this problem:

There are JSON fields named classes and classes.loaded with an integer value. ES helpfully converts the dotted notation of classes.loaded to {"classes":{"loaded":12345}}, a nested JSON object. Unfortunately, classes: 12345 cannot be parsed/indexed as classes already is an object which cannot contain a long primitive.

Obviously. there are three workarounds right now:

Feed the data to logstash and use the de_dot filter. This is computational intensive and needs a logstash server just for this use case. I would like to omit logstash as long as possible and go the route of ingest pipelines. Unfortunately, ingest pipelines do not offer anything resembling the de_dot filter.
Drop the fields that lead to problems, e.g. classes in the example given. But i have lots of those conflicting fields, and i am not fond of dropping them all. Especially as i lose the data these fields contain. Different Spring Boot services add different metrics values dynamically, calling for perpetual adaption of the list of fields to drop to get rid of the offending fields.
Using the community spring beat
GitHub - consulthys/springbeat: Simple Beat for collecting metrics from Spring Boot apps
Unfortunately, there is no binary release available, and i am not really into building this stuff myself. As i am using metricbeat anyway for system metrics, i also wanted to avoid using more independent beats for the different metrics, i would prefer a module.

The jolokia module seems to suffer from the same problem:

github.com/elastic/beats

include_fields processor does not work with dotted keys

elastic:master ← ruflin:include_fields

opened 02:37AM - 19 Dec 17 UTC

ruflin

+193 -23

The include_fields processor does not work in case the key contains a dot. See t…ests added here for this case. There are different solutions here: * Modify `GetValue` in common.MapStr to also look for keys with dots inside instead of directly walking the tree * Modifying jolokia fetcher to convert dot keys to objects The first change would be possible with adding ``` --- a/libbeat/common/mapstr.go +++ b/libbeat/common/mapstr.go @@ -296,6 +296,10 @@ func walkMap(key string, data MapStr, op mapStrOperation) (interface{}, error) { var err error keyParts := strings.Split(key, ".") + if v, exists := data[key]; exists { + return v, nil + } + ``` But that does not fully solve the problem. As in the processor the following code is used, the fields which are added after filtering to the filtered fields, will have a nesting without the docs (see Put method) ``` for _, field := range f.Fields { v, err := event.GetValue(field) if err == nil { _, err = filtered.Put(field, v) } // Ignore ErrKeyNotFound errors if err != nil && errors.Cause(err) != common.ErrKeyNotFound { errs = append(errs, err.Error()) } } ``` This Put would also cause other issues as we see in the event below. If it's convert to an object `classes` and the namespaces `classes.loaded` would conflict because one is an object and the other one isn't. I would have expected that this already causes an issue on the Elasticsearch side before filtering. ``` "jolokia":{ "metrics":{ "Metrics":{ "atomikos.nbTransactions":0.000000, "classes":18857.000000, "classes.loaded":19127.000000, "classes.unloaded":270.000000, "counter.servo.discoveryclient-httpclient_createnew":13542.000000, "counter.servo.discoveryclient-httpclient_delete":13534.000000, "uptime":406037194.000000 }, "java":{ "Threading":{ "DaemonThreadCount":37.000000, "ThreadCount":57.000000 } } } } ``` An alternative solution to the above is replacing the `.` in the names with `_` as we do for example for docker labels and other places with a similar issue. The downside of this is that filtering on fields is not straight forward as the names change between fetching from jolokia and in ES. See https://discuss.elastic.co/t/include-fields-problem/109809

The httpbeat from which the http module in metricbeat inherited already solved that problem. It had a configurable replacement of dots in the JSON field names:

github.com

christiangalsterer/httpbeat/blob/master/config/config.go#L34


      
          	BasicAuth            BasicAuthenticationConfig `config:"basic_auth"`
          	Method               string
          	Body                 string
          	Headers              map[string]string
          	ProxyUrl             string `config:"proxy_url"`
          	Timeout              *int64
          	DocumentType         string            `config:"document_type"`
          	Fields               map[string]string `config:"fields"`
          	SSL                  *outputs.TLSConfig
          	OutputFormat         string `config:"output_format"`
          	JsonDotMode          string `config:"json_dot_mode"`
          	JsonDotModeCharacter string `config:"json_dot_mode_character"`
          }
          
          type BasicAuthenticationConfig struct {
          	Username string
          	Password string
          }
          
          type ConfigSettings struct {
          	Httpbeat HttpbeatConfig

Unfortunately, the http module has no such capability.

Given the abundance of Spring Boot applications and the need for monitoring them using actuator endpoints, i am surprised there is not so much information on how to tackle the problem efficiently. Are there plans to incorporate a de_dot like capability in the http module of metricbeat, like httpbeat already had it?

Thanks,
Stefan

spfeiffer · December 19, 2017, 3:52pm

The problem also surfaced on the Spring Boot Actuator side of things:

Obviously, the problem will vanish in Spring Boot 2, but i am still bound to 1.5 unfortunately

ruflin · December 21, 2017, 4:40am

@spfeiffer Thanks for bringing this up. I agree this is a problem we need to address as it affects multiple parts in Beats. Can you open a Github issue an we can discuss there on what the best place to implement is.

system · January 18, 2018, 4:40am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
V6.0.0-beta2 http module mapping trouble Beats metricbeat	4	864	October 5, 2017
Metricbeat Http Module mapping error (Can't merge a non object mapping [http.json_namespace.classes] with an object mapping [http.json_namespace.classes] ) Beats metricbeat	3	494	February 18, 2020
HTTP-json module: processing json array Beats metricbeat	6	410	April 29, 2020
How to solve spring metrics data (in json format ) in logstash Logstash	1	503	January 25, 2019
Mapping Metricbeat to Elasticsearch Beats metricbeat	2	609	January 6, 2017

Allow for renaming of json fields in http module output

Related topics