Metricbeat decode_json_fields not working for me

Hi community,
I am new using all ELK stack and I'm recollecting data from an API with metricbeat with the http module, receiving a similar response:

    {
     "@timestamp": "2019-12-12T10:56:42.880Z",
     "event": {
      "dataset": "http.TEST",
      "duration": 237195965,
      "module": "http"
     },
     "http": {
      "TEST": {
       "data": [
        {
         "52_week_high": "385.99",
         "52_week_low": "231.23",
         "change_pct": "1.98",
        }
       ],
       "symbols_requested": 1,
      }
     },
     "metricset": {
      "name": "json",
      "period": 120000
     },
     "service": {
      "address": "apiurl.com",
      "type": "http"
     }
    } 

My problem is that I need on the output the items in "data" field at the same level of "symbols_requested" and for this I'm using the process decode_json_fields:

 - decode_json_fields:
     fields: ["data"]
     process_array: false
     max_depth: 5
     target: ""
     overwrite_keys: true
     add_error_key: false

But the data is not processed and I can't work in Kibana creating Dashboards with the data on the json

I'm using the processor correctly?

Thanks for helping me
Regards, Ivan

Hi!

decode_json_fields decodes fields containing JSON strings and your input is a normal json, so I don't think this fit in your case.

I would suggest using script processor instead so as to put the fields in the right place.

Hi! Thanks for the reply.

I'm trying the script processor, but it seems that it does not exist at least in this version of metricbeat :sleepy:

$metricbeat -e -d "*"

...
2019-12-12T17:30:11.197+0100	ERROR	instance/beat.go:916	Exiting: error initializing processors: the processor action script does not exist. Valid actions: rename, dns, add_docker_metadata, add_labels, include_fields, dissect, add_process_metadata, convert, decode_base64_field, decompress_gzip_field, drop_fields, truncate_fields, add_host_metadata, add_observer_metadata, extract_array, add_kubernetes_metadata, add_locale, community_id, add_fields, add_tags, copy_fields, decode_json_fields, drop_event, add_cloud_metadata, registered_domain

$metricbeat version
metricbeat version 7.5.0 (amd64), libbeat 7.5.0 [6d0d0ae079e5cb1d4f224801ac6df926dfb1594c built 2019-11-26 00:09:41 +0000 UTC]

And this is the configuration I have tried:

processors:
- script:
    lang: javascript
    id: my_filter
    params:
      threshold: 15
    source: >
      function process(event) {
          var d = [];
          for(var k in event.data){
              d.push(dataset[k]);
          }
      }

Regards

Hey sorry for this, it should be there. Will try to investigate.

You can use rename fields processor instead.

This one did the trick for me for system module:

processors:
  - rename:
      fields:
        - from: "system.process.memory.rss.bytes"
          to: "test_bytes"
      ignore_missing: true
      fail_on_error: false

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.