Hi all, I'm setting up Winlogbeat for collecting windows event logs and use ES and Kibana for analysis.
In my log pipeline I have Logstash in the middle as that seems to be the best current practice and allows for a central configuration and filtering, normalization etc of logs passing through. However, it seems to me like Beats is optimized to talk to directly to ES and that using Logstash (even though it is supported and not discouraged) in the pipeline will complicate things. Apart from the documentation taking for granted that Beats is talking to ES directly there is the fact that the downloadable Beats dashboard package for Kibana ships with "beats-*" indexes hardcoded (and I cannot find anywhere to change this in the Kibana UI afterwards) and hence is not really intended to support Beats transmitted through Logstash. Ofc there are ways of hacking it together either by manually editing the dashboard-files before uploading them, and manhandling stuff in Logstash into the correct indexes, but that hardly seems worth it if the logstash support will be half-ass and needing continous hacks to work.
TL;DR; Should I use Beats to ES directly since logstash support seems half-assed?
Our support for elasticsearch and logstash is the same. I'm really sorry if it doesn't feel that way, as especially for filebeat and winlogbeat logstash is a critical part.
To come back to your question: Beats with LS or ES? Obviously it depends. For Packetbeat or Metricbeat I would recommend ES directly in case you don't want to do some data enrichment, for Filebeat / Winlogbeat if you need log line parsing and data processing, use Logstash.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
