Hello everybody,
I've recently update from ES 6.8 to 7.5 and I found that Beats + Elastic pipelines are getting over Logstash (at least for ECS), and this is somehow very weird I think.
Beats should provide Logstash pipelines first, and only once they provide them they could provide an equivalent Ingest pipeline (IMHO)...otherwise the entire architecture looses its soundness.
There's a conversion tool but it's very limited: the main problem there is that provided pipelines embeds a lot of painless scripts that cannot be easily translated in Logstash (I found that only ruby scripts could be used in Logstash).
Also many Logstash filters are not compatible with ECS while beats processors and ES pipelines are..this is VERY frustrating.
Basically my question is: are beat + ingest nodes going to eventually replace Logstash (at least unofficially)??
Beats should provide Logstash pipelines first, and only once they provide them they could provide an equivalent Ingest pipeline (IMHO)...otherwise the entire architecture looses its soundness
Logstash is a very powerful tool -- it empowers our community and customers to go the "last mile" when there are no simpler solutions -- but ultimately it can be a complex tool and for many use-cases there are simpler routes that don't include Logstash. As a developer on the Logstash project, I think that acknowledging this is a Good Thing.
This is why a number of the recent solutions that have been shipped send data straight from Beats to Elasticsearch ingest node: not only are these solutions much more straight-forward to build when there are fewer moving pieces, but it is also simpler to use for many people who don't already have hardware with spare capacity running Logstash.
Elastic are continuing to invest in Logstash and the Logstash plugin ecosystem. We are investing in providing a path to ECS-compatibility in Logstash (which is very complex, considering doing so means providing a transition path that doesn't break existing pipeline configurations across thousands of human-defined pipelines and 200+ independent plugin codebases), and we are investing in stability and scale so that the users who do need Logstash can rely on it as the "stack" continues to evolve.
There are ways to use Logstash to provide secondary-enrichment, e.g., once the events have been persisted to Elasticsearch the normal way, running a Logstash pipeline that uses the Elasticsearch Input Plugin to fetch documents that haven't yet been enriched, runs them through an enrichment pipeline with whatever Logstash Filter Plugins, and outputs back into Elasticsearch with action => update.
If you'd like to open up an issue explaining what you are attempting to accomplish, we would be glad to brainstorm with you to find a solution that uses the tools at our disposal.
First of all thank you for the detailed response.
I also understand that the most straightforward way of getting things up and running is avoiding Logstash but this is why I opend this topic: this trend really push users away from Logstash.
Also the documentation is very beat-oriented and makes Logstash integration a desiderata (not a first class citizen as it should be). I think that ES processors should be a subset of Logstash filters and not the other way round. And painless scripts should be integrated into Logstash as soon as possible otherwise more and more users will eventually move away from Logstash to ES pipelines (you just need to replace Logstash with ingest nodes in the end..).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.