Filebeat's module using Logstash

Elastic Stack Beats
1

Hi,

I'm facing issue with Filebeat's module and Logstash,

If I use FileBeat and set the output to Elastic Search, everything seems to run fine. But if I output to LogStash, I do see the data in Elasticsearch but they doesn't seems to be "tagged" properly and the information doesn't appears in the Dashboard.

Here's my config file:

filebeat.yml

filebeat.modules:
- module: system
- module: auditd

#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

logstash.conf

input {
  beats {
    port => 5044
  }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
  elasticsearch {
    hosts => localhost
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

I've try to use ES output first, and then swtich to Logstash, but same issue, the data doesn't appears to be "tagged" properly. I've read the documentation and I've try the -setup -E switch, like so:
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/fileBeat -e -modules=system -setup -E "output.elasticsearch.hosts=["http://localhost:9200"]"

That seems to connect to ES first, and then use Logstash but it seems a bit hackish.

Does the Filebeat modules support Logstash or we should use ES directly?

Sorry if it's not totally clear, pretty new to ELK.

Thanks.

2

Currently FB modules only work when sending directly to Elasticsearch (because they use Ingest Node). The docs provide good overview on this.

3

Thanks.

To use LogStash, I would have to write own pipeline, correct?

4

@Elezium Correct, you can also use the ingest-convert which is shipped with Logstash to help you with the transition. See this blog post for more details

5

Also here are docs for the ingest-converter: https://www.elastic.co/guide/en/logstash/5.6/ingest-converter.html

The auditd module won't convert nicely because it uses a painless script to parse the data. Anyways I would recommend using Auditbeat to collect this data if possible.

6

Thanks both @pierhugues and @andrewkroh .

We setup FileBeat using ES directly, so far so good.

Appreciate your time.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.