Kibana visualization with two fields from prometheus data

Elastic Stack Kibana
1

Hello,

I'm not sure whether this is possible or not but really hoping for some way to use sub-buckets on the Y-axis when doing a visualization. I've got a setup where metric beat collects prometheus metrics from various hosts and what I'd like to create is a visualization of memory used ie heap - nonheap for java processes on each host.

The query for what Im attempting to graph is something along these lines:

    ((prometheus.labels.area:heap) and (prometheus.metrics.jvm_memory_bytes_used: * )) -
    ((prometheus.labels.area:nonheap) and (prometheus.metrics.jvm_memory_bytes_used: * ))

Trying to graph this however Im finding is challenging because I can't seem to be able to put the area:heap/nonheap anywhere in the visualisation

The grouping I've got on the x-axis sorted by the 'instance _name' label but I'd love to know how to select a double field when doing the graph or directly use the KQL query or if there is some other way to do this

Cheers

2

It sounds like you want to fetch two different metrics from your data set and then calculate the difference between them, is that correct?

If yes, check out the TSVB visualization - you can create a match aggregation there and enter a formula operating on other metrics for the same bucket.

3

That sounds about right, Ive used the TVSB for all the other metrics I want - this one however I cant seem to work out - I didnt see the match aggregator there but I'll have a look again,

I've tried putting in multiple datasets and in each dataset I provide the filter for on/off heap but I cant refer to them in a third dataset to subtract the two

I'll try the match and report back

4

Just noticed I have a typo in there, sorry for that (typed that on mobile). I was referring to the math aggregation.

But that will only work if the values for the two metrics are in the same document - from your latest post it seems like they are not. If that's the case the math aggregation won't help.

If that's the case could you provide example documents?

5

All good :slight_smile:

This is an example of a nonheap document:

    {
    "_index": "metricbeat-2020.03.30",
    "_type": "_doc",
    "_id": "v4BQKXEBGpOTx6FYOGo2",
    "_version": 1,
    "_score": null,
    "_source": {
        "@version": "1",
        "agent": {
        "version": "7.6.1",
        "type": "metricbeat",
        "id": "5b1bd7dd-a898-4462-82e8-3f1aece57a1a",
        "hostname": "bba233076a95",
        "ephemeral_id": "5daad9a9-49e2-489c-9b38-e3a8eeed95ff"
        },
        "service": {
        "type": "prometheus",
        "address": "rpcnode:9545"
        },
        "event": {
        "module": "prometheus",
        "dataset": "prometheus.collector",
        "duration": 7623138
        },
        "prometheus": {
        "metrics": {
            "jvm_memory_bytes_max": -1,
            "jvm_memory_bytes_committed": 72376320,
            "jvm_memory_bytes_used": 68069744,
            "jvm_memory_bytes_init": 7667712
        },
        "labels": {
            "instance": "rpcnode:9545",
            "job": "prometheus",
            "area": "nonheap"
        }
        },
        "type": "metricbeat",
        "@timestamp": "2020-03-30T02:40:54.727Z",
        "metricset": {
        "period": 15000,
        "name": "collector"
        },
        "ecs": {
        "version": "1.4.0"
        },
        "host": {
        "name": "bba233076a95"
        }
    },
    "fields": {
        "@timestamp": [
        "2020-03-30T02:40:54.727Z"
        ]
    },
    "sort": [
        1585536054727
    ]
    }

The field Im intrested in is prometheus.metrics.jvm_memory_bytes_used and the second field to sort is prometheus.labels.area == "nonheap"

The heap memory documents are the same, with the exception of the labels.are = heap

    {
    "_index": "metricbeat-2020.03.30",
    "_type": "_doc",
    "_id": "xIBQKXEBGpOTx6FYOGo2",
    "_version": 1,
    "_score": null,
    "_source": {
        "@version": "1",
        "agent": {
        "type": "metricbeat",
        "version": "7.6.1",
        "id": "5b1bd7dd-a898-4462-82e8-3f1aece57a1a",
        "hostname": "bba233076a95",
        "ephemeral_id": "5daad9a9-49e2-489c-9b38-e3a8eeed95ff"
        },
        "service": {
        "type": "prometheus",
        "address": "rpcnode:9545"
        },
        "event": {
        "module": "prometheus",
        "dataset": "prometheus.collector",
        "duration": 7730852
        },
        "prometheus": {
        "metrics": {
            "jvm_memory_bytes_max": 8355053568,
            "jvm_memory_bytes_committed": 832569344,
            "jvm_memory_bytes_used": 470165304,
            "jvm_memory_bytes_init": 522190848
        },
        "labels": {
            "instance": "rpcnode:9545",
            "job": "prometheus",
            "area": "heap"
        }
        },
        "type": "metricbeat",
        "@timestamp": "2020-03-30T02:40:54.727Z",
        "metricset": {
        "period": 15000,
        "name": "collector"
        },
        "ecs": {
        "version": "1.4.0"
        },
        "host": {
        "name": "bba233076a95"
        }
    },
    "fields": {
        "@timestamp": [
        "2020-03-30T02:40:54.727Z"
        ]
    },
    "sort": [
        1585536054727
    ]
    }

And I would like to graph the difference of the two values, I think Timelion may be my only option here?

6

Ah, I see. Yeah, in that case I think using timelion with the .subtract function is a good option:
.es(q="prometheus.labels.area:heap",metric=sum:prometheus.metrics.jvm_memory_bytes_used).subtract(.es(q="prometheus.labels.area:nonheap",metric=sum:prometheus.metrics.jvm_memory_bytes_used))

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.