Hi,
I'm a new user to Elastic and in order to better understand how elastic works, i would like to create a visualisation or canvas that would allow me to compare per process memory usage on my server.
More used to SQL language, i've decied to do it through a line chart in a canvas.
I'm using elastic stack 7.12 and I get data from metricbeats with ECS v1.8
At the moment i've managed to get data i need as so :
SELECT "@timestamp" AS date, "process.pid" as "pid", "process.memory.pct" as "pct" FROM "metricbeat-*" WHERE "pid" IS NOT NULL
Using this request i have all datas i want but they're not "sorted" the way i want.
I don't know if it's possible or not but i would like to have something like
[{pid1 : [{time1, pct1_1}, {time2, pct1_2}, ... ]}, {pid2: {time1, pct2_1}, {time2, pct2_2}, ... ]}]
In the end i would like to have a chart that would look kind of this :
As i mentioned earlier, i'm only trying to step up and learn, if this is not possible through an SQL request i can switch to whatever allow me to do so
I might suggest try a Lens visualization if you're just learning Elasticsearch and Kibana you should be able to make that graph and just a couple minutes. It's very flexible and user friendly.
I'm not sure if using SQL is really your requirement or just something you're used to but I would start with Lens it's a little less complex than Canvas.
I'm not a fan of drag and drop things i did give it a try at the begining but didn't feel really comfortable with it.
Using SQL is not a requirement but as a developer, I had some experiments with SQL and felt more comfortable at the beginning.
Whatever thanks for the advice, i'm going to try again with Lens.
Also something I forgot to mention, i'm using a Self Hosted Basic Licence
You were right it's kind of easy to do it with Lens i wasn't aware of everything first time i tried. I'll try to reproduce it in a custom visualisation now, some things with Lens feel like thew will not be enough fully customise it the way I want. For example i didn't found a way to have data per process. i need to either split them by user or by pid but if i split them by pid they Lens create a range which can leads to a non accurate visualisation. If i have process 1 and 2 which takes 30 and 1 % of memory usage, lens show me a 31% memory usage on the pid range between 1 and 10 and this is not accurate enough for my usage
is there a way to see how data are processed maybe the requests made by the lens tool or anything like that ? I think it could help me better understand what happen backend and so create a TSVB visualisation that fully fits my requierments
Thanks by advance
Right now Lens does not have the Inspect button which shows the request and response.
If you want to explore the data I would highly recommend looking at Discover to see the raw data and then of course in Dev Tools you can query the data using the DSL or SQL
Yeah I used the break down by pid but as you can see it create ranges and not 4000 -> 4999, 5000 -> 5999 (this i what i meant here :
I can understand why it does such a thing but it does not fit with PID because in my opinion you want to know the exact pid of the process that uses a lot of memory. But it's such a special case that you cannot manage this and make it simple to use. moreover, There are so many process at the same time on a system this can make the visualisation really hard to read.
I'm trying to create something that would best fit my usecase in a TSVB visualisation atm
Alright no problem I think this could be a great addition in the future for people who wants to create their custom visualisations with very specific requierements.
I do know about the discover page thanks I was wondering if this feature exists in order to understand the way data gets fetch more than just looking at row data
Edit : I know this is a bit off topic but is there a way to retrive other data than {{value}} in a TSVB Data Formatter template ?
Ohh I stand corrected hold on a minute let me check!!!!
It is because pids are numbers not keywords therefore they are treated like numbers if you do process.name (which is a distinct term / value you will see what you expect.
Let me check on the pids.... yeah that is going to be a problem because pid are are longs (insert long discussion here on difference between keywords / terms and numerics and cardinality)
So I do not think you are going to get what you want.
Process.name doesn't work either. if i have two process from java (for example elasticsearch and logstash) they will both appear as java but they aren't from the same process and so on their memory usage is not the same. (i've tried it before )
And as you mentioned pids are numbers and so on Lens try to make range out of them.
i managed to make something that fits my request using a TSVB but i'm runnign into UX issue aha
Yeah this look like what i was trying to archieve thanks
I have approximatly the same on TSVB so i'll try to continue from my TSVB visualisation and fix my UX issue (for example if take the above example you made, i'll now try to have something like process.name(process.pid) from user.name : process.memory.rss.bytes
But as you mentionned this is not the place i'll open a new topic if i do not manage to do it myself ^^
Huge thanks for the help overall, learned a lot thanks to your advices
have a great day
EDIT : Idk how to close the topic, so i'll just leave it open but the problem has been fix
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
I think this could be a great addition in the future for people who wants to create their custom visualisations with very specific requierements.
I was wondering if this feature exists in order to understand the way data gets fetch more than just looking at row data
)
