I work for a major IT services enterprise that builds analytics solutions for our clients. We have aggressively dug our heals into leveraging/using ElasticSearch and Kibana in places where other RDBMS + Web-SQL-Visualize technologies stacks are used and deeply rooted today.
I see a cliff we are going to fall off by having such 'total' focus on ES/Kibana for things that are outside time-series visualizations and machine learning requirements... looking for advice...
You may start out enabling ES/Kibana for visualization of time-series data in a very powerful way.. and see wonderful things in metrics/logs realtime and be able to visualize that and do Watcher and ML......but:
(a) then the customer demands the ability to incorporate all the other styles of reporting and visualization that are not graphic report visualizations and not simple ( Kibana Table ) formats of information. They want reports, they want interfaces that are not the 'out of box' graph types of Kibana.
(b) Very concerning.... we find you are ingesting relational data that normalized in very well done schemas....and now we are "flattening" out say 2,3,4,5 and more tables of data and creating incredibly redundant data structures because of the "one index" constraints that all data must be in one index pattern (similar documents/indices) or one index for a report. I see this non scalable
(c) Users are coming up with relationship/context associations that are not designed in these 'flat' indices above.. so we are going back and rebuilding and reloading indices... Not agile from a user standpoint at all.... they are needing to run through our IT shop and not empowered to run their own SQL joins and report creations...
Has anyone from Elastic team or any public SME published any well-written reports or standards/guidelines on how clients should be using ElasticSearch/Kibana in "complement" to other RDBMS datastores and RDBMS-accessing-portal-technologies ?
My organization is "all in" on ElasticSearch/Kibana but what i see is that there is a huge space where RDBMSs and RDBMS-accessing-portal-technologies should continue to be used and are not "appropriate" for Kibana/ES... but there is an architecture framework for both to coexist.
Has Elastic published any such guidance ? Can it be a topic at ElasticCon ? Does anyone have any links to any such collateral ? Without seeing really much out there from Elastic, we are going to have to go build the architecture decisions/framework guidance ourselves.
Thank you for your time everyone.