Conflicted about Beats, Logstash and ES


(Stian Ovrevage) #1

Hi all, I'm setting up Winlogbeat for collecting windows event logs and use ES and Kibana for analysis.

In my log pipeline I have Logstash in the middle as that seems to be the best current practice and allows for a central configuration and filtering, normalization etc of logs passing through. However, it seems to me like Beats is optimized to talk to directly to ES and that using Logstash (even though it is supported and not discouraged) in the pipeline will complicate things. Apart from the documentation taking for granted that Beats is talking to ES directly there is the fact that the downloadable Beats dashboard package for Kibana ships with "beats-*" indexes hardcoded (and I cannot find anywhere to change this in the Kibana UI afterwards) and hence is not really intended to support Beats transmitted through Logstash. Ofc there are ways of hacking it together either by manually editing the dashboard-files before uploading them, and manhandling stuff in Logstash into the correct indexes, but that hardly seems worth it if the logstash support will be half-ass and needing continous hacks to work.

TL;DR; Should I use Beats to ES directly since logstash support seems half-assed? :slight_smile:


Beats via Logstash and Elasticsearch
(ruflin) #2

Our support for elasticsearch and logstash is the same. I'm really sorry if it doesn't feel that way, as especially for filebeat and winlogbeat logstash is a critical part.

Even if you send filebeat data through Logstash, we recommend to use the filebeat index. If you use our default config options documented here, this should be the case: https://www.elastic.co/guide/en/beats/libbeat/1.2/logstash-installation.html#logstash-setup So also dashboards should work.

For the dashboard loading, you should find some more documents here: https://www.elastic.co/guide/en/beats/libbeat/1.2/dashboard-load-options.html

This PR here should make it also possible to change the "hardcoded" index pattern, but it is not well documented yet: https://github.com/elastic/beats/pull/1439

To come back to your question: Beats with LS or ES? Obviously it depends. For Packetbeat or Metricbeat I would recommend ES directly in case you don't want to do some data enrichment, for Filebeat / Winlogbeat if you need log line parsing and data processing, use Logstash.


(Mark Walkom) #3

Why do you say this?


(system) #4

This topic was automatically closed after 21 days. New replies are no longer allowed.