I'm struggling with Canvas performance on a Raspberry Pi 4. My data are stored in Elastic every second to the server (i7-5500U, 16 Go RAM). Around 1500 metric elements in my Canvas should be ideally updated at the same time through an individual SQL query. Since I only need the very last data for each element, I guess optimizations can be done. Today, it takes 2 min 30 to update the Canvas on the RPI 4 and 7 seconds on my laptop (i5-8350U CPU).
What do you think about it ? Where should I look ?
Thank you for your help ! I confirm that each element has its own SQL query. My application is a battery supervision. I made a script to generate the whole .json Canvas file.
My main element is an "image reveal" based on the battery SoC (State of Charge) like a fuel gauge.
SELECT CAST(bms.soc AS FLOAT)/100 AS Soc FROM "batterylogs-bms-2020.*" WHERE position=3 AND rack=1 AND site='invalides' AND box=1 AND time > NOW() - INTERVAL 10 SECONDS ORDER BY time DESC LIMIT 1
On the top, I use a simple metric element to display the SoC as a numeric value. In addition, other elements such as a charging icon, is overlaid on the battery image.
SELECT CAST(ds.flags.charging AS INTEGER) AS charging FROM "batterylogs-ds-2020.*" WHERE position=3 AND rack=1 AND site='invalides' AND box=1 AND time > NOW() - INTERVAL 10 SECONDS ORDER BY time DESC LIMIT 1
Elasticsearch, Logstash and Kibana : 7.6.1
My RPi is connected to a wifi hotspot and my server is directly connected to the internet router by ethernet. Both are on the same network.
When I load this huge Canvas from my laptop, it already takes several seconds to edit the workpad settings like the title (queries are idled). It's barely possible to enable the refresh period on the RPi so much the Canvas is slowed. The performance is proportional to the number of elements.
On the other hand, the Kibana discover menu loads my data instantly on my laptop and takes few seconds on the RPi. Thus my first guess is that the bottleneck lies in the Canvas which is great and simple to handle by the way.
the Kibana discover menu loads my data instantly on my laptop and takes few seconds on the RPi
This is not surprising to me -- part of the reason you are seeing a performance difference in Discover and Canvas is because both are using completely different querying infrastructures to handle talking to Elasticsearch. Canvas has a single expression backing each element which is going to initiate its own request, whereas Discover is generally going to send a single request to retrieve the data.
When I load this huge Canvas from my laptop, it already takes several seconds to edit the workpad settings like the title (queries are idled). It's barely possible to enable the refresh period on the RPi so much the Canvas is slowed. The performance is proportional to the number of elements.
My guess is that the biggest hurdle you are hitting here is number of requests -- each of those elements is likely triggering its own HTTP request. Even if batched those could end up taking quite some time.
Unfortunately I'm not sure I know of a good away around this, other than developing your own Kibana plugin which does more aggressive request batching and perhaps combines each of the 1500 elements.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.