Filebeat's module using Logstash

Hi,

I'm facing issue with Filebeat's module and Logstash,

If I use FileBeat and set the output to Elastic Search, everything seems to run fine. But if I output to LogStash, I do see the data in Elasticsearch but they doesn't seems to be "tagged" properly and the information doesn't appears in the Dashboard.

Here's my config file:

filebeat.yml

filebeat.modules:
- module: system
- module: auditd

#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

logstash.conf

input {
  beats {
    port => 5044
  }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
  elasticsearch {
    hosts => localhost
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

I've try to use ES output first, and then swtich to Logstash, but same issue, the data doesn't appears to be "tagged" properly. I've read the documentation and I've try the -setup -E switch, like so:
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/fileBeat -e -modules=system -setup -E "output.elasticsearch.hosts=["http://localhost:9200"]"

That seems to connect to ES first, and then use Logstash but it seems a bit hackish.

Does the Filebeat modules support Logstash or we should use ES directly?

Sorry if it's not totally clear, pretty new to ELK.

Thanks.

Currently FB modules only work when sending directly to Elasticsearch (because they use Ingest Node). The docs provide good overview on this.

Thanks.

To use LogStash, I would have to write own pipeline, correct?

@Elezium Correct, you can also use the ingest-convert which is shipped with Logstash to help you with the transition. See this blog post for more details

Also here are docs for the ingest-converter: https://www.elastic.co/guide/en/logstash/5.6/ingest-converter.html

The auditd module won't convert nicely because it uses a painless script to parse the data. Anyways I would recommend using Auditbeat to collect this data if possible.

Thanks both @pierhugues and @andrewkroh .

We setup FileBeat using ES directly, so far so good.

Appreciate your time.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.