Hello. I need to setup a service showing logs from 10+ domains. I would like to filter the log data by a domain and the data range. It would be nice to setup user accounts (user A can access logs from a domain A', user B can access only the data from a domain B').
My team decided the best option is to use the ELK Stack.
So far, we accomplished the sending logs part. Now we are sending the data to central point (using Beats and Logstash). And this part is working well.
But now, my team claims it's the best option (faster and cheaper) to resign from ElasticSearch and do a custom solution for the log storage and presentation. They would like to send logs from Logstash to the relational DB. And then, they would like to build the presentation layer in React.
I am not a part of the IT team, that's why I ask you. Are they right? Is a custom solution more time and money savvy? Or should they reconsider using ElasticSearch and Kibana?
@Tomly, I am sure you realize that on this forum you will get a lot of opinions that are slanted toward the Elastic Stack. But let me break down your question into a few areas...
There is a reason why your IT team are asking the question about whether or not to move away from the Elastic Stack. If they are new to the stack it is likely because of some frustration they are experiencing related to their knowledge gap with the technology. This is not a criticism on your team. The simple fact is that the Elastic Docs are more of a reference material. There is a serious lack of HOW-TO material that discusses anything beyond the basics (that's about as deep as Elastic training gets you). I have felt this pain myself, as have most who have gone before you. The good news is that you have a thriving community that you can reach out to for help. Once you persevere through the learning curve, you will find that there are things that you can do with the Elastic Stack, that you would struggle to achieve with more traditional toolsets.
Many organizations want help mastering the stack, or simply would like someone to come in and do the work for them, and they are willing to pay for that help. In that case there are a number of independent consultants and services companies that can help you out. This is what me and my small team do (sorry for the shameless plug, but I think it is relevant to your question). So what I am saying is don't be afraid to find some help. Two examples, that may be interesting to you:
We recently worked with an organization that had been trying for 18 months to create their own security log monitoring solution. Leveraging the Elastic Stack we were able to do for them in 7 days, what they couldn't do on their own in 18 months.
Another organization we worked with was collecting IoT metrics related to vehicle traffic. We took the requirements for the MVP of their new product, and delivered the solution (based on Elastic Stack) in 2 days, which included the deployment of a multi-node cluster in Azure. We are now engaged to rearchitect their entire cloud-based data platform and the Elastic Stack will play a critical role.
The subject of your message asks "is it good to implement the ELK Stack without Elastic"? I assume that you are referring to whether or not you need one of the paid subscriptions. It really depends on your requirements. It is hard to give concrete guidance without understanding your use-case better, but here are a few quick thoughts (but purely my opinion)...
X-Pack security, is usually the way to go if you need that feature set (and many people do).
X-Pack Machine Learning can be a powerful addition once you have data collection mastered, and can focus on analytics.
I am not a fan of X-Pack Graph. I just never found a compelling use-case for it, and I find its usefulness hindered by the fact that it can't placed on a dashboard.
X-Pack Alerting is fine, but there are other options that can work equally well, in some cases even better (depends on your overall architecture). Of course if you have X-Pack Gold for security, you also have Alerting so why not use it.
Getting back to your team. They may be thinking about building their own solution from scratch because they want a more customized experience. Fortunately the Elastic Stack provides the opportunity to create plugins, such as custom visualizations for Kibana. There is a lot of opportunity for your team to get creative and innovative and make the solutions more of their own. So why not focus their time on that? And leave the problems that have already been solved to the Elastic Stack?
Hopefully this provides you with a few things to think about.
I mean, my team would like to resign from Elasticsearch and Kibana. I've just updated the title.
So, they are inclined to this type of stack: Beats (done) + Logstash (done) + relational database + front end in React. So, I understand it this way: a relational database instead of Elasticsearch. And the Custom solution in React instead of Kibana. They believe it's better and faster,
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.