I've been trying and failing do a good job visualizing data with Kibana, so I'm looking for advice.
First, all my indices on which I produce visualizations have two important properties that I often want to group by in reports:
- Deployment (e.g. "US-EAST-1")
- Firm (e.g. "Pizza Eaters, Inc.")
Most of the reports in my Kibana (where "report is usually a visualization, but might also be a search in some cases) will want to group by deployment and firm properties.
Then there's the stuff in the records that are all tagged with the above that I need to report on. For example:
There's more, but the above 5 things should suffice.
So here's my problem. Let's say I want to have a report on "Documents Uploaded by User",
in two flavors, one grouped by project, one that isn't.
Here are some of the problems I face in rendering these things in Kibana.
If I want a visualization grouped by all 5 fields above, some visualizations like bar graphs have a limit of 4, so I can't do that. (And of course I probably shouldn't do that, trying to squeeze too many facets into a visualization makes for a pretty unreadable visualization).
I need to show data for ALL firms and ALL deployments. But I can't seem to split the charts enough to do that. If I was reporting on projects there could be hundreds of projects. How can I enumerate ALL things in a dashboard or other grouped page of data?
Sometimes I'd like to ... scroll ... a visualization. For example, sometimes I put in a "publish time" visualization that shows when key record sets were deposited to Elasticsearch and allows users to select timeframes by clicking on the visualizaiton. Only there's no way to zoom out once you've zoomed in, and there's no way to scroll the visualation right or left (forward/backward in time). Scrolling might solve other problems where there's more data in the visualization than can be reported. (E.g., how about scrolling from matters with most documents to matters with fewest documents?)
Using a search in a dashboard is rarely useful if only because I can only sort on one column. So if I sort, say, on Deployment, I can't sub-sort on the other interesting things like firm/matter/user. The "Data Table" visualization suffers from the same problem.
I love the visualization capabilities. But I need to have a better way to allow my users to drill down by deployments, firms, or other facets, and potentially see the full extent of the facet they're examining. E.g. all projects, and at that point I feel I am lost with Kibana.
If there were a Kibana API I suppose I could create a dashboard for each deployment and populate a visualization for each firm. Or generate all the reports I might want and graft links to them into another portal that allows users to pick what they want, since Kibana Dashboards are fairly heavy and clumsy things to navigate in the Kibana portal. At the end of the day perhaps Kibana is just my visualization generating tool.
Or maybe I shouldn't be using Kibana at all? It was originally viewed as our quick "in" to time based aggregation of things, but I need to aggregate by other things too and my initial portal implemented in Kibana leaves a lot to be desired.
I'm also reading up on plugins, though I don't think I've really answered my questions about "what can a plugin do for me" with respect to the issues I'm facing.