Get per process Metrics

I'm a new user to Elastic and in order to better understand how elastic works, i would like to create a visualisation or canvas that would allow me to compare per process memory usage on my server.
More used to SQL language, i've decied to do it through a line chart in a canvas.
I'm using elastic stack 7.12 and I get data from metricbeats with ECS v1.8

At the moment i've managed to get data i need as so :

SELECT "@timestamp" AS date, "" as "pid", "process.memory.pct" as "pct" FROM "metricbeat-*" WHERE "pid" IS NOT NULL  

Using this request i have all datas i want but they're not "sorted" the way i want.
I don't know if it's possible or not but i would like to have something like
[{pid1 : [{time1, pct1_1}, {time2, pct1_2}, ... ]}, {pid2: {time1, pct2_1}, {time2, pct2_2}, ... ]}]

In the end i would like to have a chart that would look kind of this :

As i mentioned earlier, i'm only trying to step up and learn, if this is not possible through an SQL request i can switch to whatever allow me to do so

Any help would be appreciate. Thanks by advance.

Hi @Dzious Welcome to the community!

Good work so far.

I might suggest try a Lens visualization if you're just learning Elasticsearch and Kibana you should be able to make that graph and just a couple minutes. It's very flexible and user friendly.

I'm not sure if using SQL is really your requirement or just something you're used to but I would start with Lens it's a little less complex than Canvas.

Just a thought.

I'm not a fan of drag and drop things i did give it a try at the begining but didn't feel really comfortable with it.
Using SQL is not a requirement but as a developer, I had some experiments with SQL and felt more comfortable at the beginning.
Whatever thanks for the advice, i'm going to try again with Lens.

Also something I forgot to mention, i'm using a Self Hosted Basic Licence

It was just a suggestion.

Basic license is fine you should be able to do lens or canvas or any visualization.

If you really want to use canvas if you install the sample data there will be sample canvases I think you could look at and see the SQL / sample data.

Lens is becoming a powerful visualization tool it does derivatives moving averages etc.

There is also another visualization tool called TSVB it is the metrics power tool you can also look at that

You were right it's kind of easy to do it with Lens i wasn't aware of everything first time i tried. I'll try to reproduce it in a custom visualisation now, some things with Lens feel like thew will not be enough fully customise it the way I want. For example i didn't found a way to have data per process. i need to either split them by user or by pid but if i split them by pid they Lens create a range which can leads to a non accurate visualisation. If i have process 1 and 2 which takes 30 and 1 % of memory usage, lens show me a 31% memory usage on the pid range between 1 and 10 and this is not accurate enough for my usage

is there a way to see how data are processed maybe the requests made by the lens tool or anything like that ? I think it could help me better understand what happen backend and so create a TSVB visualisation that fully fits my requierments
Thanks by advance

The Break down by will split by the etc

Right now Lens does not have the Inspect button which shows the request and response.

If you want to explore the data I would highly recommend looking at Discover to see the raw data and then of course in Dev Tools you can query the data using the DSL or SQL

Yeah I used the break down by pid but as you can see it create ranges and not 4000 -> 4999, 5000 -> 5999 (this i what i meant here :

I can understand why it does such a thing but it does not fit with PID because in my opinion you want to know the exact pid of the process that uses a lot of memory. But it's such a special case that you cannot manage this and make it simple to use. moreover, There are so many process at the same time on a system this can make the visualisation really hard to read.
I'm trying to create something that would best fit my usecase in a TSVB visualisation atm

Alright no problem :slight_smile: I think this could be a great addition in the future for people who wants to create their custom visualisations with very specific requierements.
I do know about the discover page thanks :smiley: I was wondering if this feature exists in order to understand the way data gets fetch more than just looking at row data

Edit : I know this is a bit off topic but is there a way to retrive other data than {{value}} in a TSVB Data Formatter template ?

Ohh I stand corrected hold on a minute let me check!!!!

It is because pids are numbers not keywords therefore they are treated like numbers if you do (which is a distinct term / value you will see what you expect.

Let me check on the pids.... yeah that is going to be a problem because pid are are longs (insert long discussion here on difference between keywords / terms and numerics and cardinality)

So I do not think you are going to get what you want.

Here is a pic with process name.

If you want to talk about TSVB formatters you should open a separate thread with an appropriate subject we prefer not to mixed threads.

Haha on Me I stand Corrected Again!! here it is ...

You can use Top Values! doesn't work either. if i have two process from java (for example elasticsearch and logstash) they will both appear as java but they aren't from the same process and so on their memory usage is not the same. (i've tried it before :smiley: )

And as you mentioned pids are numbers and so on Lens try to make range out of them.
i managed to make something that fits my request using a TSVB but i'm runnign into UX issue aha

Yeah i will later on sry ^^

Wow this looks great let me try this :wink:

See previous response use top values on

TSVB is awesome it just has a bit of a learning curve.

But I did get what you want in lens it's right there

Yeah this look like what i was trying to archieve thanks :smiley:
I have approximatly the same on TSVB so i'll try to continue from my TSVB visualisation and fix my UX issue (for example if take the above example you made, i'll now try to have something like from : process.memory.rss.bytes
But as you mentionned this is not the place i'll open a new topic if i do not manage to do it myself ^^
Huge thanks for the help overall, learned a lot thanks to your advices
have a great day :slight_smile:

EDIT : Idk how to close the topic, so i'll just leave it open but the problem has been fix

1 Like

Your welcome it will close on its own.
Ask more questions / Have fun!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.