Boundaries - When to use Kibana/ES and when NOT to use

[lbi2000]

I work for a major IT services enterprise that builds analytics solutions for our clients. We have aggressively dug our heals into leveraging/using ElasticSearch and Kibana in places where other RDBMS + Web-SQL-Visualize technologies stacks are used and deeply rooted today.

I see a cliff we are going to fall off by having such 'total' focus on ES/Kibana for things that are outside time-series visualizations and machine learning requirements... looking for advice...

Opinion...

You may start out enabling ES/Kibana for visualization of time-series data in a very powerful way.. and see wonderful things in metrics/logs realtime and be able to visualize that and do Watcher and ML......but:
(a) then the customer demands the ability to incorporate all the other styles of reporting and visualization that are not graphic report visualizations and not simple ( Kibana Table ) formats of information. They want reports, they want interfaces that are not the 'out of box' graph types of Kibana.
(b) Very concerning.... we find you are ingesting relational data that normalized in very well done schemas....and now we are "flattening" out say 2,3,4,5 and more tables of data and creating incredibly redundant data structures because of the "one index" constraints that all data must be in one index pattern (similar documents/indices) or one index for a report. I see this non scalable
(c) Users are coming up with relationship/context associations that are not designed in these 'flat' indices above.. so we are going back and rebuilding and reloading indices... Not agile from a user standpoint at all.... they are needing to run through our IT shop and not empowered to run their own SQL joins and report creations...

Has anyone from Elastic team or any public SME published any well-written reports or standards/guidelines on how clients should be using ElasticSearch/Kibana in "complement" to other RDBMS datastores and RDBMS-accessing-portal-technologies ?

My organization is "all in" on ElasticSearch/Kibana but what i see is that there is a huge space where RDBMSs and RDBMS-accessing-portal-technologies should continue to be used and are not "appropriate" for Kibana/ES... but there is an architecture framework for both to coexist.

Has Elastic published any such guidance ? Can it be a topic at ElasticCon ? Does anyone have any links to any such collateral ? Without seeing really much out there from Elastic, we are going to have to go build the architecture decisions/framework guidance ourselves.

Thank you for your time everyone.

[VijayDoshi]

Hi,

Thanks for your post, I’ll do my best to answer your questions, I’m the Product Lead for Kibana.

Your question hits on an issue that I’ve spent a good portion of my career working with companies to address. Before joining Elastic I spent more than seven years approaching this from the other side of the spectrum working for a business intelligence vendor where the conversation was often ‘at what point do relational databases and tools that connect into them reach their limit and non-relational data technologies (like Elastic - specifically Elasticsearch and Kibana) become necessary to achieve success?’ Having seen both sides of this issue now, I can safely say there is no definitive line or best practice.

Each customer I speak to has their own balance when it comes to data technologies and while some may favor relational data products for the majority of their use cases, still others like your organization will see products like Elasticsearch and Kibana providing the capabilities they need to handle things like data analysis, reporting, dashboarding, etc. No one will claim that you only need one though which is why I am so glad to see you use the word “complement” in your question since that is the approach that always works best. For Elastic’s products specifically, this suits customers well since robust flexibility is a key part of our design principal largely inherited from our open source roots. It not only lets our customers use things like Elasticsearch and Kibana for use cases most non-relational data technologies wouldn’t be brought into, it also means that Elastic plays alongside relational solutions well too (more specifics on our ongoing efforts there below). Now let me address some of the specifics of your questions...

(a) We see this as a really important shift too and we are working towards making it easier to perform Ad-Hoc analysis right in Kibana. Lens beta will be available in our next release (7.5) and is a perfect example of the direction we are headed to support drag and more types of users - keep your eye out for Lens! In addition, we are working on improvements to Elastic SQL such that you can connect third party BI tools to your Elasticsearch instance for analysis. Ultimately, we want to make it really easy to visualize and interact with the data in Elasticsearch via Kibana. Regarding viz types; over time, we’ll add more visualization types; however, there is quite a bit you can do within the product. Can you let me know what specific visualizations they are interested in seeing? Can the missing visualizations be accomplished using Vega, Canvas or in any of the Solutions we offer?

(b) Flattening the data is the expected pattern in Elasticsearch; we don’t see this changing in the near-term. There are tremendous advantages of our architecture as it enables the distributed scalability of Elasticsearch via sharding; and two, we explicitly have features around optimizing storage (frozen indices & rollups). Flattening and transforming data at ingest vs. joining/scripting fields on the fly is performance and scalability; when you survey other models this difference quickly jumps out. In Lens, you will be able to pick data from multiple index patterns; Canvas also allows you to use multiple index patterns and in Canvas you can use Elastic SQL to query your data - I’d recommend exploring those - Canvas you can get today and soon Lens will be available.

(c) We hear this pretty often and are working towards making it easier to ingest and transform data. This is an area of investment for us; however, it will take some time to make it a delightful experience that an end-user could interact with. We do have nested queries as a solution to joins.

As far as reference materials - there are patterns; however, every organization seems to have a unique ecosystem & requirements they are trying to manage; I suspect this diversity is one reason why there is not a single reference. I know within Elastic we pull a whole slew of business data sources into Elasticsearch and use Kibana on top of it. Most of these systems start as RDBMs or we access via APIs, it has many billions of documents. Here is a webinar that might be helpful.

We really appreciate your input! I’ll poke around for some additional articles or blogs too.

Vijay