Metricbeat decode_json_fields not working for me

[Ivan_Vazquez (Iván Vázquez)]

Hi community,
I am new using all ELK stack and I'm recollecting data from an API with metricbeat with the http module, receiving a similar response:

    {
     "@timestamp": "2019-12-12T10:56:42.880Z",
     "event": {
      "dataset": "http.TEST",
      "duration": 237195965,
      "module": "http"
     },
     "http": {
      "TEST": {
       "data": [
        {
         "52_week_high": "385.99",
         "52_week_low": "231.23",
         "change_pct": "1.98",
        }
       ],
       "symbols_requested": 1,
      }
     },
     "metricset": {
      "name": "json",
      "period": 120000
     },
     "service": {
      "address": "apiurl.com",
      "type": "http"
     }
    } 

My problem is that I need on the output the items in "data" field at the same level of "symbols_requested" and for this I'm using the process decode_json_fields:

 - decode_json_fields:
     fields: ["data"]
     process_array: false
     max_depth: 5
     target: ""
     overwrite_keys: true
     add_error_key: false

But the data is not processed and I can't work in Kibana creating Dashboards with the data on the json

I'm using the processor correctly?

Thanks for helping me
Regards, Ivan

[ChrsMark (Chris Mark)]

Hi!

decode_json_fields decodes fields containing JSON strings and your input is a normal json, so I don't think this fit in your case.

I would suggest using script processor instead so as to put the fields in the right place.

[Ivan_Vazquez (Iván Vázquez)]

Hi! Thanks for the reply.

I'm trying the script processor, but it seems that it does not exist at least in this version of metricbeat

$metricbeat -e -d "*"

...
2019-12-12T17:30:11.197+0100	ERROR	instance/beat.go:916	Exiting: error initializing processors: the processor action script does not exist. Valid actions: rename, dns, add_docker_metadata, add_labels, include_fields, dissect, add_process_metadata, convert, decode_base64_field, decompress_gzip_field, drop_fields, truncate_fields, add_host_metadata, add_observer_metadata, extract_array, add_kubernetes_metadata, add_locale, community_id, add_fields, add_tags, copy_fields, decode_json_fields, drop_event, add_cloud_metadata, registered_domain

$metricbeat version
metricbeat version 7.5.0 (amd64), libbeat 7.5.0 [6d0d0ae079e5cb1d4f224801ac6df926dfb1594c built 2019-11-26 00:09:41 +0000 UTC]

And this is the configuration I have tried:

processors:
- script:
    lang: javascript
    id: my_filter
    params:
      threshold: 15
    source: >
      function process(event) {
          var d = [];
          for(var k in event.data){
              d.push(dataset[k]);
          }
      }

Regards

[ChrsMark (Chris Mark)]

Hey sorry for this, it should be there. Will try to investigate.

You can use rename fields processor instead.

This one did the trick for me for system module:

processors:
  - rename:
      fields:
        - from: "system.process.memory.rss.bytes"
          to: "test_bytes"
      ignore_missing: true
      fail_on_error: false