How To?: Visualizations with a lot of dimensions

Elastic Stack Kibana
1

I've been trying and failing do a good job visualizing data with Kibana, so I'm looking for advice.

First, all my indices on which I produce visualizations have two important properties that I often want to group by in reports:

  1. Deployment (e.g. "US-EAST-1")
  2. Firm (e.g. "Pizza Eaters, Inc.")

Most of the reports in my Kibana (where "report is usually a visualization, but might also be a search in some cases) will want to group by deployment and firm properties.

Then there's the stuff in the records that are all tagged with the above that I need to report on. For example:

  1. User
  2. Project
  3. Document

There's more, but the above 5 things should suffice.

So here's my problem. Let's say I want to have a report on "Documents Uploaded by User",
in two flavors, one grouped by project, one that isn't.

Here are some of the problems I face in rendering these things in Kibana.

  1. If I want a visualization grouped by all 5 fields above, some visualizations like bar graphs have a limit of 4, so I can't do that. (And of course I probably shouldn't do that, trying to squeeze too many facets into a visualization makes for a pretty unreadable visualization).

  2. I need to show data for ALL firms and ALL deployments. But I can't seem to split the charts enough to do that. If I was reporting on projects there could be hundreds of projects. How can I enumerate ALL things in a dashboard or other grouped page of data?

  3. Sometimes I'd like to ... scroll ... a visualization. For example, sometimes I put in a "publish time" visualization that shows when key record sets were deposited to Elasticsearch and allows users to select timeframes by clicking on the visualizaiton. Only there's no way to zoom out once you've zoomed in, and there's no way to scroll the visualation right or left (forward/backward in time). Scrolling might solve other problems where there's more data in the visualization than can be reported. (E.g., how about scrolling from matters with most documents to matters with fewest documents?)

  4. Using a search in a dashboard is rarely useful if only because I can only sort on one column. So if I sort, say, on Deployment, I can't sub-sort on the other interesting things like firm/matter/user. The "Data Table" visualization suffers from the same problem.

I love the visualization capabilities. But I need to have a better way to allow my users to drill down by deployments, firms, or other facets, and potentially see the full extent of the facet they're examining. E.g. all projects, and at that point I feel I am lost with Kibana.

If there were a Kibana API I suppose I could create a dashboard for each deployment and populate a visualization for each firm. Or generate all the reports I might want and graft links to them into another portal that allows users to pick what they want, since Kibana Dashboards are fairly heavy and clumsy things to navigate in the Kibana portal. At the end of the day perhaps Kibana is just my visualization generating tool.

Or maybe I shouldn't be using Kibana at all? It was originally viewed as our quick "in" to time based aggregation of things, but I need to aggregate by other things too and my initial portal implemented in Kibana leaves a lot to be desired.

I'm also reading up on plugins, though I don't think I've really answered my questions about "what can a plugin do for me" with respect to the issues I'm facing.

Advice?

2

From how I understand what you are trying to do, I'd say that, to start with, ignore the Deployment and Firm metrics in the visualizations themselves, and instead use the query bar to filter down the records. If you create visualizations that cover ALL Deployments and Firms, you only need to worry about the tags, so you may have just solved #1, and since they show everything, you have definitely solved #2.

So now what about dealing with those criteria? Well, if your dashboard has all of the visualizations and searches that you want to see, you can then use the query bar with a query like this: Deployment:"US-EAST-1" AND Firm:"Pizza Eaters, Inc.", and just modify those as needed. You can also do this with the filter bar, which may be even easier since you can just toggle them on and off instead of typing them in. Either way, a common use pattern is to take whatever combos you need and save them as their own dashboards to make it easier to users to simply select what they want. I'll admit that's not great, but it does work.

Since everything is just a record in the .kibana index in Elasticsearch, you could script the creation of these things as well. If you take a look at an existing saved Dashboard document, you'll see that there's not a lot of information there. Once you have 1 working example, creating the rest is just a matter of writing new documents and changing the query in them.

As for #3, that's a neat idea. There's an open issue to add next/prev controls to the timepicker, so you could step through your data with the click of a button, and it sounds like that would cover some of what you need. "Zooming" back out you can actually do already with the back button in your browser, it's just not obvious that it works that way. I encourage you to open another issue to discuss adding the something to handle your most/fewest use case.

Lastly, the way you describe #4, it sounds more like what you want to do is filter on a column, and not sort on it. It sounds like you are trying to sort to find the Deployment you want, and then sort on another column to arrange the data in that Deployment, and as you note, that's not possible. You can, however, click on that Deployment and it will filter the data to match, then you can sort the result using another field. Forgive me if I'm misunderstanding what you are trying to do and got that wrong though.

There was a lot to cover here, and hopefully I touched on it all, but let me know if I missed anything, or if something wasn't clear.

3

Thanks for the suggestions Joe, all very good. The problem with filters and typing them in is that my user base consists of people who may not know what they want to filter for, and definitely won't want to learn any syntax for typing stuff in. They just want to bookmark their favorite charts or dashboards and click in.

FYI, one thing I do is create a data table visualization of deployments and firms and call it the 'deployment and firm selector' with a help markdown widget next to it explaining what it's for. It's one way to select in from among the deployments and firms, though you'll have to scroll to the top of the dashboard and confirm or otherwise tweak the filters, something which is slightly unwieldy both from the UI standpoint and as I find the dashboards to be a strain on the browser. (Heck, for that matter, I notice loading a single dashboard can blow out the queue lengths on the Elasticsearch server depending on number of shards and things being queried from the dashboard - these Dashboards are demanding!).

Well, I'll be getting a lot more feedback from my users in the coming month, the Kibana portal I've released is just going out to production. We'll see how they respond to the workarounds I've used and whether they're willing to type in the query string bar. I was just trying to see if there wasn't a way I missed to generate reports for all possible deployments so they don't have to select for it and/or add/remove filters. I'll look at the scripting angle too.