# Group Element through query to use in Vega visualisation **URL:** \ **Category:** Kibana\ **Tags:** vega\ **Created:** [April 16, 2021, 12:51pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389 "2021-04-16T12:51:54Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Dzious](https://avatars.discourse-cdn.com/v4/letter/d/0ea827/32.png) [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Post date:** [April 16, 2021, 12:51pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/1 "2021-04-16T12:51:54Z")
Hi, I'm currently learning about query and stuffs. In order to do so I aim to have a vega visualisation that would allow me to visualise a per process memory usage. So far, i managed to do it using both Lens and TSVB. Unfortunately they did not met all my requierements (or at least i didn't manange to find a way to build the visualisation i wanted) This lead to my aim to create a Vega visualisation. I managed to get data I need but unfortunately i didn't manage to group the as i wanted. My question is how can i group my data. Thanks to this query : ```auto "query": { "exists": { "field": "process.pid" } }, "fields": [ "@timestamp", "process.pid", "system.process.memory.rss.bytes", "process.name", "user.name" ], "_source": false ``` I get this kind of data : ```auto { "took": 45, "timed_out": false, "_shards": { "total": 13, "successful": 13, "skipped": 11, "failed": 0 }, "hits": { "total": 128594, "max_score": 1, "hits": [ { "_index": ".ds-metricbeat-cl01ptocor00-dev-elastic_stack-2021.04.14-000001", "_type": "_doc", "_id": "_aUX0XgBDu3u7xL3cpWD", "_score": 1, "fields": { "system.process.memory.rss.bytes": [ 568008704 ], "process.name": [ "java" ], "@timestamp": [ "2021-04-14T15:54:37.043Z" ], "user.name": [ "elasticsearch" ], "process.pid": [ 26131 ] } }, // Lots other hits ] } } ``` my target would be to have something like : ```auto { "took": 45, "timed_out": false, "_shards": { "total": 13, "successful": 13, "skipped": 11, "failed": 0 }, "hits": { "total": 128594, "max_score": 1, "hits": [ { "process.pid" : 26131, "process.name": "java" "user.name": "elasicsearch", "memory_usage": [ { "@timestamp" : "2021-04-14T15:54:37.043Z", "bytes": 568008704 }, // metrics on other timestamps ] } // other process ] } } ``` I really do not understand how i am supposed to do so and some help would be appreciate. Thanks by advance ---
**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\ **Post date:** [April 18, 2021, 4:34pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/2 "2021-04-18T16:34:39Z")
A little confused but I will take a stab at it. I think you want to do something like this instead. ```auto "query": { "exists": { "field": "process.pid" } }, "_source": [ "@timestamp", "process.pid", "system.process.memory.rss.bytes", "process.name", "user.name" ] ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 18, 2021, 5:35pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/3 "2021-04-18T17:35:47Z")
Hi @Dzious I see you are progressing... Curious could you describe the visualization you are looking for? Did you see this ... this is a 2 level Tree Map. If you use a KQL filter on `user.name` at the top it would be 3 level 🙂 ![Screen Shot 2021-04-18 at 10.28.53 AM](https://us1.discourse-cdn.com/elastic/original/3X/4/b/4be8a5d6d94cc89bcc5231176a33c2d87eceaeaf.png) If you are looking for a 3 Level Tree Map say User / Process Name / PID average with Process Memory. I already built one in Vega It is made to be a 3 level Tree Map with a Value .. .Like Disk Space / CPU etc. Perhaps take a look it is at this [here](https://github.com/bvader/howtos/tree/master/vega-tree-map) ... bvader is me... Use at your own risk 🙂 It has been a long time...... ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 18, 2021, 5:42pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/4 "2021-04-18T17:42:34Z")
And By the way here is a line chart Split By Username, Process Name, PID with Process memory values. ![Screen Shot 2021-04-18 at 10.45.25 AM](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b47483c9b6309a2f2ef7cea6219134ac8084cd13.png) This is the line chart under ![Screen Shot 2021-04-18 at 10.09.49 AM](https://us1.discourse-cdn.com/elastic/original/3X/0/6/0654716ebace382775e39950d9dbb7eda5836831.png) ![Screen Shot 2021-04-18 at 10.09.55 AM](https://us1.discourse-cdn.com/elastic/original/3X/3/3/338038eb8a721efe4dbb6320e9f840b642a5175f.png) Don't let me talk you out of Vega, Vegas is fun but there's definitely a learning curve. ---
**Author:** ![Dzious](https://avatars.discourse-cdn.com/v4/letter/d/0ea827/32.png) [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Post date:** [April 19, 2021, 3:11pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/5 "2021-04-19T15:11:59Z")
Hi, Thanks for you reply. After my post i kept trying new things and the solution you gave me was a step in my learning experience. Unfortunately this was still not the data structure i wanted. With this solution i still have my memory usage / timestamp in differents hits. My aim is to group them into one hit per process which contains all my memory usage/timestamp But thanks for trying to help me 😃 ---
**Author:** ![Dzious](https://avatars.discourse-cdn.com/v4/letter/d/0ea827/32.png) [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Post date:** [April 19, 2021, 3:25pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/6 "2021-04-19T15:25:00Z")
Hi @stephenb Yeah I am progressing 🙂 The visualisation i aim to it the exact same as the one you show on your other reply 😃 I don't quite understand which `...` you're refering to could try tell me a bit more ? 😅 I will have a look at it thanks. Even if i do not use it i'll learn things that can only help me 😃 Even if i leave Vega for the moment i still have a lot of visualisation planned to do and i think that i will need it for some of them. I do not abandon Vega but i need to go ahead onto other type of coniguration for the moment. Thanks for the help you brougth me along this first learning experience ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 19, 2021, 3:37pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/7 "2021-04-19T15:37:33Z")
> [@stephenb](#): > > Perhaps take a look it is at this [here](https://github.com/bvader/howtos/tree/master/vega-tree-map) ... bvader is me... > > Use at your own risk 🙂 It has been a long time...... sorry just the way I type this is a sample, I wrote with a lot of help, it has been a long time since I looked at it, and I don't use Vega that often so I am not the best resource and it could have bugs in it. I am `bvader` on github some people are confused by that. If you work in vega I would use the vega editor it is very helpful it can be found [here](https://vega.github.io/editor/#/) Good luck on your journey... ---
**Author:** ![Dzious](https://avatars.discourse-cdn.com/v4/letter/d/0ea827/32.png) [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Post date:** [April 20, 2021, 5:50am UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/8 "2021-04-20T05:50:12Z")
No problem mate 😃 yeah i had a look at it yesterday before leaving and this seems bit too complicated on the first sight x) I'll eventually go back to it later on when i'll be building more visualisations ^^ I already use vega, this helped me a lot during my first hours of experiencing. Thanks for the tip anyway ^^ Thanks ! But don't fell free right away, I think we'll see each other soon 😛 have a great day 🙂 ---
**Author:** ![Dzious](https://avatars.discourse-cdn.com/v4/letter/d/0ea827/32.png) [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Post date:** [April 20, 2021, 11:58am UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/9 "2021-04-20T11:58:20Z")
Well I eventually have a question for you Stephen, How did you managed to get your values in Mib, I do not manage to find it. 😅 Thanks by advance ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 20, 2021, 1:40pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/10 "2021-04-20T13:40:31Z")
Values in MiB in which solution / visualization? ---
**Author:** ![Dzious](https://avatars.discourse-cdn.com/v4/letter/d/0ea827/32.png) [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Post date:** [April 20, 2021, 1:56pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/11 "2021-04-20T13:56:18Z")
On this screenshot : ![Screen Shot 2021-04-18 at 10.45.25 AM](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b47483c9b6309a2f2ef7cea6219134ac8084cd13.png) On the left hand side memory usage is set as Mib On what seems to be the same visualisation i have this ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/6/9/6972038d54f9e67a9e81dc20feaf2e761ffdb604.png) As you can see, my memory usage is set as byte and not Mib nor Gib ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 20, 2021, 2:30pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/12 "2021-04-20T14:30:14Z")
So assuming you ran `metricbeat setup` correctly with metricbeat that should happen automatically so I have a small concern there but you can check and fix. Are your sure you ran metricbeat setup correctly also I am a little confused as some places seems like you are using metricbeat and others seem like you are using the elasticagent. Example I am not sure where you got this index.... it just may be my lack of understanding... > [@Dzious](#): > > ```auto > "_index": ".ds-metricbeat-cl01ptocor00-dev-elastic_stack-2021.04.14-000001", > > ``` Anyways .... Go To Stack Management / Index Patterns / metricbeat-\* Mine already had the format set... and the should as part of the module ![Screen Shot 2021-04-20 at 7.23.07 AM](https://us1.discourse-cdn.com/elastic/original/3X/9/9/99b6fde6c0d75cca8487b4de28ac27ffdfd9bc90.png) If not edit it and set it... It should be set for you as part of the module... ![Screen Shot 2021-04-20 at 7.23.16 AM](https://us1.discourse-cdn.com/elastic/original/3X/3/0/3065955cc917a00eaea8238c02d21438e47a9ca8.png) ---
**Author:** ![Dzious](https://avatars.discourse-cdn.com/v4/letter/d/0ea827/32.png) [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Post date:** [April 21, 2021, 7:09am UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/13 "2021-04-21T07:09:05Z")
That worked thanks 😃 Also i'm using Metricbeat and I installed it through the rpm package from [elastic.co](http://elastic.co) download page. I'll double check it is installed correctly but i think so. About the index, this come from the fact that i am using Data Stream to make the setup easier. Anyway that not the topic 😉 Once again thanks for your help ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 21, 2021, 1:07pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/14 "2021-04-21T13:07:33Z")
Even though you install through rpm you still need run setup after you update your configs. You only need to run it once total no matter 1 or 1000 hosts, just run it first. `metricbeat setup -e` meyr ---
**Author:** ![Dzious](https://avatars.discourse-cdn.com/v4/letter/d/0ea827/32.png) [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Post date:** [April 21, 2021, 1:42pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/15 "2021-04-21T13:42:03Z")
Alright thanks. Unforutnately I've got an error due to Elasticsearch output not enable (i use Logstash) This topic is not about metricbeat installation. Shall i make a new topic for this ? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 21, 2021, 4:12pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/16 "2021-04-21T16:12:41Z")
To run setup just temporarily point filebeat output at elasticsearch. Then when setup's over point filebeat output back to logstash that's actually the process. And as I said that only needs to happen once so once you've done that you can just leave the elasticsearch output portion commented out. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 19, 2021, 4:13pm UTC](https://discuss.elastic.co/t/group-element-through-query-to-use-in-vega-visualisation/270389/17 "2021-05-19T16:13:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.