Kibana + Vega: Visualize CPU performance log


I have a problem. I have a assignment at work to visualize some CPU performance data with Kibana and Vega-lite.
The data is coming from a log file.
Each row i the log file looks something like this:
It's a coma separated field where the first value is time(some internal time inside the software) in milliseconds and fifth field the cpu usage in percentage at that time.

Each log file is from a separate run using our software and each row a sample of cpu usage from that run.

The log file is being read by filebeat and sent to logstash.

I have a pipeline in logstash that separates the message field using grok.
So the output from logstash that is sent to elasticsearch looks something like this

{ "_index": "filebeat-2018.03.28", "_type": "doc", "_id": "XV4ObWIByRQFAhnbcioL", "_score": 1, "_source": { "cpu_run_exe": "1.340000e+01", "adpestatus": "0", "host": "TMGLSE6TYYGH2", "@timestamp": "2018-03-28T14:41:24.361Z", "source": """c:\filebeat\logs\PERFORMANCELOG19011349-FULL.TXT""", "cpu_post_run_exe": "1.340000e+01", "beat": { "version": "6.2.3", "hostname": "TMGLSE6TYYGH2", "name": "TMGLSE6TYYGH2" }, "read_missed_events": "2", "write_missed_events": "0", "sysctrlstatus": "2", "message": "1294086,0,2,1.340000e+01,1.340000e+01,1.340000e+01,1.340000e+01,2,0,54,48", "post_run_exe_missed_events": "48", "time": "1294086", "@version": "1", "prospector": { "type": "log" }, "cpu_read_io": "1.340000e+01", "tags": [ "beats_input_codec_plain_applied" ], "run_exe_missed_events": "54", "offset": 33443, "cpu": "1.340000e+01" } }

What I want to do with this data is to visualize it using Vega-lite and display the cpu usage over time.
And also I want to be able to see the difference in cpu usage between two or more logs.

So the X-axis in my Vega graph should display the time from the logfile and the Y-axis should show the
CPU usage.

I have never encountered Kibana or Vega before - so I'm having a hard time grasping how this would be done.

I hope this can be done in some way and some suggestions would be appreciated!

To make it clear. The question is about how to code Vega to display this data.

No reason against it, but what is the reason that you have to use Vega? Have you tried any of the other visualization forms to create this graph?

@johanyeriksson while possible, you might have a fairly steep learning curve. You need to get the data into an ES server, write an ES query to get the data you need, and to learn Vega-Lite (or Vega) language to create the graph. You might want to see vega-lite tutorial, and read up the Vega-related blog posts.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.