# Kibana visualization with two fields from prometheus data

**URL:** <https://discuss.elastic.co/t/kibana-visualization-with-two-fields-from-prometheus-data/225099>\
**Category:** Kibana\
**Created:** [March 26, 2020, 2:41am UTC](https://discuss.elastic.co/t/kibana-visualization-with-two-fields-from-prometheus-data/225099 "2020-03-26T02:41:08Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jf11](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jf11](https://discuss.elastic.co/u/jf11)\
**Post date:** [March 26, 2020, 2:41am UTC](https://discuss.elastic.co/t/kibana-visualization-with-two-fields-from-prometheus-data/225099/1 "2020-03-26T02:41:09Z")

</div>

Hello,

I'm not sure whether this is possible or not but really hoping for some way to use sub-buckets on the Y-axis when doing a visualization. I've got a setup where metric beat collects prometheus metrics from various hosts and what I'd like to create is a visualization of memory used ie heap - nonheap for java processes on each host.

The query for what Im attempting to graph is something along these lines:

```auto
    ((prometheus.labels.area:heap) and (prometheus.metrics.jvm_memory_bytes_used: * )) -
    ((prometheus.labels.area:nonheap) and (prometheus.metrics.jvm_memory_bytes_used: * )) 

```

Trying to graph this however Im finding is challenging because I can't seem to be able to put the area:heap/nonheap anywhere in the visualisation

The grouping I've got on the x-axis sorted by the 'instance \_name' label but I'd love to know how to select a double field when doing the graph or directly use the KQL query or if there is some other way to do this

Cheers

---

<div class="post-metadata">

**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\
**Post date:** [March 26, 2020, 9:45am UTC](https://discuss.elastic.co/t/kibana-visualization-with-two-fields-from-prometheus-data/225099/2 "2020-03-26T09:45:40Z")

</div>

It sounds like you want to fetch two different metrics from your data set and then calculate the difference between them, is that correct?

If yes, check out the `TSVB` visualization - you can create a match aggregation there and enter a formula operating on other metrics for the same bucket.

---

<div class="post-metadata">

**Author:** ![jf11](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jf11](https://discuss.elastic.co/u/jf11)\
**Post date:** [March 27, 2020, 3:52am UTC](https://discuss.elastic.co/t/kibana-visualization-with-two-fields-from-prometheus-data/225099/3 "2020-03-27T03:52:55Z")

</div>

That sounds about right, Ive used the TVSB for all the other metrics I want - this one however I cant seem to work out - I didnt see the match aggregator there but I'll have a look again,

I've tried putting in multiple datasets and in each dataset I provide the filter for on/off heap but I cant refer to them in a third dataset to subtract the two

I'll try the match and report back

---

<div class="post-metadata">

**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\
**Post date:** [March 27, 2020, 1:57pm UTC](https://discuss.elastic.co/t/kibana-visualization-with-two-fields-from-prometheus-data/225099/4 "2020-03-27T13:57:18Z")

</div>

Just noticed I have a typo in there, sorry for that (typed that on mobile). I was referring to the `math` aggregation.

But that will only work if the values for the two metrics are in the same document - from your latest post it seems like they are not. If that's the case the `math` aggregation won't help.

If that's the case could you provide example documents?

---

<div class="post-metadata">

**Author:** ![jf11](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jf11](https://discuss.elastic.co/u/jf11)\
**Post date:** [March 30, 2020, 2:46am UTC](https://discuss.elastic.co/t/kibana-visualization-with-two-fields-from-prometheus-data/225099/5 "2020-03-30T02:46:53Z")

</div>

All good 🙂

This is an example of a nonheap document:

```
    {
    "_index": "metricbeat-2020.03.30",
    "_type": "_doc",
    "_id": "v4BQKXEBGpOTx6FYOGo2",
    "_version": 1,
    "_score": null,
    "_source": {
        "@version": "1",
        "agent": {
        "version": "7.6.1",
        "type": "metricbeat",
        "id": "5b1bd7dd-a898-4462-82e8-3f1aece57a1a",
        "hostname": "bba233076a95",
        "ephemeral_id": "5daad9a9-49e2-489c-9b38-e3a8eeed95ff"
        },
        "service": {
        "type": "prometheus",
        "address": "rpcnode:9545"
        },
        "event": {
        "module": "prometheus",
        "dataset": "prometheus.collector",
        "duration": 7623138
        },
        "prometheus": {
        "metrics": {
            "jvm_memory_bytes_max": -1,
            "jvm_memory_bytes_committed": 72376320,
            "jvm_memory_bytes_used": 68069744,
            "jvm_memory_bytes_init": 7667712
        },
        "labels": {
            "instance": "rpcnode:9545",
            "job": "prometheus",
            "area": "nonheap"
        }
        },
        "type": "metricbeat",
        "@timestamp": "2020-03-30T02:40:54.727Z",
        "metricset": {
        "period": 15000,
        "name": "collector"
        },
        "ecs": {
        "version": "1.4.0"
        },
        "host": {
        "name": "bba233076a95"
        }
    },
    "fields": {
        "@timestamp": [
        "2020-03-30T02:40:54.727Z"
        ]
    },
    "sort": [
        1585536054727
    ]
    }

```

The field Im intrested in is prometheus.metrics.jvm\_memory\_bytes\_used and the second field to sort is prometheus.labels.area == "nonheap"

The heap memory documents are the same, with the exception of the labels.are = heap

```
    {
    "_index": "metricbeat-2020.03.30",
    "_type": "_doc",
    "_id": "xIBQKXEBGpOTx6FYOGo2",
    "_version": 1,
    "_score": null,
    "_source": {
        "@version": "1",
        "agent": {
        "type": "metricbeat",
        "version": "7.6.1",
        "id": "5b1bd7dd-a898-4462-82e8-3f1aece57a1a",
        "hostname": "bba233076a95",
        "ephemeral_id": "5daad9a9-49e2-489c-9b38-e3a8eeed95ff"
        },
        "service": {
        "type": "prometheus",
        "address": "rpcnode:9545"
        },
        "event": {
        "module": "prometheus",
        "dataset": "prometheus.collector",
        "duration": 7730852
        },
        "prometheus": {
        "metrics": {
            "jvm_memory_bytes_max": 8355053568,
            "jvm_memory_bytes_committed": 832569344,
            "jvm_memory_bytes_used": 470165304,
            "jvm_memory_bytes_init": 522190848
        },
        "labels": {
            "instance": "rpcnode:9545",
            "job": "prometheus",
            "area": "heap"
        }
        },
        "type": "metricbeat",
        "@timestamp": "2020-03-30T02:40:54.727Z",
        "metricset": {
        "period": 15000,
        "name": "collector"
        },
        "ecs": {
        "version": "1.4.0"
        },
        "host": {
        "name": "bba233076a95"
        }
    },
    "fields": {
        "@timestamp": [
        "2020-03-30T02:40:54.727Z"
        ]
    },
    "sort": [
        1585536054727
    ]
    }

```

And I would like to graph the difference of the two values, I think Timelion may be my only option here?

---

<div class="post-metadata">

**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\
**Post date:** [March 30, 2020, 7:53am UTC](https://discuss.elastic.co/t/kibana-visualization-with-two-fields-from-prometheus-data/225099/6 "2020-03-30T07:53:52Z")

</div>

Ah, I see. Yeah, in that case I think using timelion with the `.subtract` function is a good option:  
`.es(q="prometheus.labels.area:heap",metric=sum:prometheus.metrics.jvm_memory_bytes_used).subtract(.es(q="prometheus.labels.area:nonheap",metric=sum:prometheus.metrics.jvm_memory_bytes_used))`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2020, 7:54am UTC](https://discuss.elastic.co/t/kibana-visualization-with-two-fields-from-prometheus-data/225099/7 "2020-04-27T07:54:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.