Canvas Blog Stream

(Rashid Khan) #41

Partial expressions are no more!

You may remember partial expressions as those things you pass into mapColumn. Now all sub-expressions appear equal, no need to lead with a $ anymore. The function’s argument definition now determines if the interpreter should resolve the sub-expression with the current context or if the interpreter should provide an instance of itself to the function to execute with whatever context it needs. We also get type checking out of this, which makes the plugin interface even more consistent than before.

This is exceptionally awesome for conditional functions as it tightly controls execution paths. Previously execution paths that would never be returned had to execute anyway. This makes expressions with lots of logic execute much faster, and more reliably.

What does this mean to you? Not much, just stop using $.


demodata | mapColumn random fn=${math 'random()'}


demodata | mapColumn random fn={math 'random()'}

We just ignore the $ for now, but you really should update your expressions, because we're going to remove the backward compatibility with that syntax before Beta 1. Tick tock.

(Rashid Khan) #42

More layout engine sneak peaks

We're reviewing the code for this now. Party time.....


(Joe Fleming) #43

SQL support is here

SQL Access has landed in X-Pack with the 6.3.0 release that happened today. You saw it used in Canvas at Elasticon, and it's pretty much as awesome as it looked. And now, as of Canvas preview release 0.1.2002, which just went out, you can finally use it yourself!

It's not quite as polished as we'd like, but it's available for use right now. Some quick notes:

  • You must map your data using the pointseries function to use it with pie or plot, even if the data is aggregated in the SQL query already.
  • If you are doing math operations in the SQL query, such as avg(bytes), you should alias the result (avg(bytes) as bytes, for example), otherwise using the value may result in expression errors.
  • You can only query simple fields in Elasticsearch. If you try to query fields in Elasticsearch that have array values, as one example, SQL will throw an error.

Head on over to the Elastic downloads page to grab version 6.3.0 of the stack, and consult the Canvas installation guide to get Canvas installed.

(Rashid Khan) #44

Stability is good

Currently Canvas runs all of it's server tasks in the main process. This is all well and good when things are going, uh, well and good, but what about when it all falls apart?

That's what we're working on right now: Running everything we can in a nice, safe, sub-process that we can monitor and slay if it appears to be misbehaving. We have a working prototype, but shipping it is still a ways off. I will warn you, the plugin API will probably change a bit. Function definitions should stay the same, roughly, but the way to register them will probably need to change. We're still sorting out exactly what that will look like.

(Rashid Khan) #45

Maybe this preview mode will become a thing

I can't promise this will make it in, but check out this neat preview mode:

(Rashid Khan) #46

Canvas, your window to Kibana 7

Maybe you know this, maybe you don't, but you're keeping up to date on Canvas so I'm guessing you're pretty interested in the future: Kibana is getting a big pretty re-skinning for version 7. It's not like it's a secret. In fact, you can watch the design language progress over at The EUI project page.

It isn't merged yet, but here's what Canvas looks like with the K7 styling applied

We should have that merged soon!

(Rashid Khan) #47

Ship Shape

Canvas 2050 is out and includes this little shape element which makes it easier to compose together custom looking elements. See...


(Alex Francouer) #49

We're quite happy to present our first guest Canvas blog post! Edan Shahmoon from Micro Focus was kind enough to share his experience with Canvas. Have a similar story you'd like to share? Feel free to tweet me at @alexf_elastic.


While watching the World Cup games, some questions came to mind. Are there any patterns for the timing of goals? Does the number of fouls affect the chances to win or lose? As well as many other questions that a professional football player won’t ask- but we will!

I chose Elasticsearch for this project for its capability to analyze time series data and Kibana to visualize the data and to transform it into knowledge. I also heard about Canvas and I found this project as an opportunity to experiment with it. So, let’s get started!

Data Preparation

The data source for this project is “world_cup_json” - a JSON based REST API for the world cup statistics.

When I found this API, I thought that the ETL process will be a piece of cake. I’ll just have to upload these JSONs into Elasticsearch and thanks to the dynamic mapping of Elasticsearch, my data will be ready for analysis.

I created a Python script which implements this logic, prepared a Dockerfile to handle the dependencies of this script elegantly and it worked!

I could see all the data in Elasticsearch although it wasn’t good enough. In the raw data there were lists (or multilevel objects) that I would like to analyze as a standalone document.

Normalization – Since my data is structured and relatively flat, I decided to be “conservative” and to transform it as if I would transform it for a traditional SQL database. I decomposed the original documents into smaller documents which represent the entities:

  • Matches – General information about a match (location, weather, teams...)
  • Team stats – The statistics of each team at each match (goals, corners, off sides…)
  • Events – information about events like goals, tickets and substitutions.

These steps eventually helped me to analyze the data with the new SQL feature.

Create artificial timestamps for Timelion – At the raw data the events (goals, yellow tickets, substitutions…) are represented as a list. In addition to each event, there was a field showing the time that the event occurred. That field was formatted as the minute that the event occurred. I wanted to use the time series analysis features of Timelion, so I decided to break these lists into standalone documents and to set a timestamp for each event. To compare events from different matches,I mapped the first minute of all the matches to the first minute of 2018. Thanks to this trick, I could use all the features Timelion offered.

For example:

After we loaded the data into Elasticsearch, we can begin to analyze the data in Canvas!

Visualize with Canvas

For this project, I decided to try out Canvas. It’s highly customizable and very promising. As with every innovation, it takes some time to get used to it.

In the first page of my workpad I used different elements of Canvas to show some statistics about the final match of the World Cup:

Here I used Markdown and the repeatImage to show the statistics in a user-friendly way.

I asked myself if the ball possession can help to predict the winner. It wasn’t significant as you can see at the final match. My hypothesis was that the reason for this is the high number of goals that came from penalties. I figured out that actually about 25% of the goals in the world cup came from penalties or own goals which aren’t affected by the ball possession!

The second page shows the distribution of goals over time:

The third one shows the distribution of substitutions over time:

Want to analyze the World Cup data by yourself?

The project is available on GitHub. I created a Docker compose file which set up everything automatically. The README file provides all the technical details, so head on over to try it out!

About the Author

Edan Shahmoon is a SaaS DevOps Engineer at Micro Focus. He has been a part of a team that is responsible for Elasticsearch logging clusters and helps gain insights from logs with Kibana. Edan is passionate about data mining, monitoring, Elasticsearch or any combination of them!

(Rashid Khan) #50

Hey, bozo, where are the releases?

On the way. We've got a big release sitting under the heat lamp right now. The deal is, like you've seen in previous posts, we've applied all these awesome new Kibana 7 styles. Except, Kibana 6.3.x doesn't jive with them, so we're waiting for 6.4.0 to go out. Once it does, we'll cut a release with all the awesome new stuff, including the layout engine. Check it out: elements now snap to other elements along edges and center axises. It even works well at rotated corners of elements. This can be disabled with the meta key (Command on Mac)

(Rashid Khan) #51

New Elements

Also, there are a bunch of new elements coming down the pipe. More below the fold on this screenshot too. Including the much request "metric" element.

(Rashid Khan) #52

Canvas for Kibana 6.4.0

Kibana 6.4.0 is live, and so is Cavnas 2174 for Kibana 6.4.0. This is the first release with the fancy new Kibana 7 styling! It also has all those new elements, that cool new layout engine (with snapping and alignment!) and even a handy-dandy new csv function.

(Rashid Khan) #53

Improved reporting

Did you know Canvas supports PDF export? Yep, integrates with Watcher too if that's your thing! You'll need support for Chromium enabled in kibana:

        type: chromium

We released 2198 a few minutes ago with lots of bug fixes and this handy-dandy new export dialog!

(Rashid Khan) #54

Also, no more red boxes

Ok, still some red boxes, but you have to click to see them. They're not all in-yo-face like they were. Elements with errors are now this soothing triangle. Click the triangle to get the details

(Rashid Khan) #55

Really? Steam?

Alex made this cool page showing large buildings that still run on steam. I'll let him elaborate if he has time, but here's a screenshot...

(Rashid Khan) #56

Behold, the progress unicorn

Look, I know I do this "hey, looks what's coming" thing a lot. But if you weren't interested in what's coming you probably wouldn't be here, right? We've gotten a bunch of requests for a "gauge" element. And that's all well and good, we're doing that. We have a bar, circle, semi-circle, you know gauges, but we wanted to add some fun options. Behold the progress unicorn. Coming soon.

(Rashid Khan) #57

There was a webinar

Sorry I didn't tip you off to this earlier, but we hosted a SQL webinar and did the whole thing in Canvas. Check it out the recording over here:

(Rashid Khan) #58

More example workpads.

Alex has been hacking away on Canvas stuff and made this totally rad github repo of examples: Some are specific to datasets, while others teach you something about how Canvas works. Like this one that shows how you can conditionally show images:

(Rashid Khan) #59

Simplifying Canvas server plugins

By now you probably know that we work pretty hard to keep Canvas plugins simple. For example, we don't do any wacky transpilation stuff for browser code. That means Canvas plugins don't go through the Kibana "Optimizer" and are wicked fast to install.

While tracking down a bug in Canvas we decided to bring that same philosophy to our server code (see this pull comment). Before this change, if Kibana changed something in how it builds server code, you'd be on the hook to sort it out. Ew.

So now the Canvas server opts out of Kibana server side transpilation. Bringing in a package? Use require(). Or if you really want, use babel to get back import statements. You certainly don't have to use babel (heck, I probably wouldn't for most projects), but you can.

Most importantly, this change brings additional assurances of plugin API stability, and keeps our technology choices out of your way!