Hi community,
I am new using all ELK stack and I'm recollecting data from an API with metricbeat with the http module, receiving a similar response:
{
"@timestamp": "2019-12-12T10:56:42.880Z",
"event": {
"dataset": "http.TEST",
"duration": 237195965,
"module": "http"
},
"http": {
"TEST": {
"data": [
{
"52_week_high": "385.99",
"52_week_low": "231.23",
"change_pct": "1.98",
}
],
"symbols_requested": 1,
}
},
"metricset": {
"name": "json",
"period": 120000
},
"service": {
"address": "apiurl.com",
"type": "http"
}
}
My problem is that I need on the output the items in "data" field at the same level of "symbols_requested" and for this I'm using the process decode_json_fields:
- decode_json_fields:
fields: ["data"]
process_array: false
max_depth: 5
target: ""
overwrite_keys: true
add_error_key: false
But the data is not processed and I can't work in Kibana creating Dashboards with the data on the json
I'm using the processor correctly?
Thanks for helping me
Regards, Ivan
ChrsMark
(Chris Mark)
December 12, 2019, 2:03pm
2
Hi!
decode_json_fields
decodes fields containing JSON strings and your input is a normal json, so I don't think this fit in your case.
I would suggest using script processor instead so as to put the fields in the right place.
Hi! Thanks for the reply.
I'm trying the script processor, but it seems that it does not exist at least in this version of metricbeat
$metricbeat -e -d "*"
...
2019-12-12T17:30:11.197+0100 ERROR instance/beat.go:916 Exiting: error initializing processors: the processor action script does not exist. Valid actions: rename, dns, add_docker_metadata, add_labels, include_fields, dissect, add_process_metadata, convert, decode_base64_field, decompress_gzip_field, drop_fields, truncate_fields, add_host_metadata, add_observer_metadata, extract_array, add_kubernetes_metadata, add_locale, community_id, add_fields, add_tags, copy_fields, decode_json_fields, drop_event, add_cloud_metadata, registered_domain
$metricbeat version
metricbeat version 7.5.0 (amd64), libbeat 7.5.0 [6d0d0ae079e5cb1d4f224801ac6df926dfb1594c built 2019-11-26 00:09:41 +0000 UTC]
And this is the configuration I have tried:
processors:
- script:
lang: javascript
id: my_filter
params:
threshold: 15
source: >
function process(event) {
var d = [];
for(var k in event.data){
d.push(dataset[k]);
}
}
Regards
ChrsMark
(Chris Mark)
December 13, 2019, 8:47am
4
Hey sorry for this, it should be there. Will try to investigate.
You can use rename fields
processor instead.
This one did the trick for me for system
module:
processors:
- rename:
fields:
- from: "system.process.memory.rss.bytes"
to: "test_bytes"
ignore_missing: true
fail_on_error: false
system
(system)
Closed
January 10, 2020, 8:47am
5
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.