Multiple filebeat output to ES and Logstash

[mathur7vidit (vidit mathur)]

Team, i have a requirement where i need to send audit, auth and syslog from servers to elasticsearch directly and application's log to logstash. here are the changes and steps i performed. I am using ubuntu 16 in our setup.

1. after installation of filebeat. i replicated /etc/filebeat folder as /etc/filebeat1.
2. then did necessary changes in filebeat.yml as below.

filebeat.config.prospectors:
enabled: true
path: /etc/filebeat/conf.d/*.yml
reload.enabled: true
reload.period: 5s

setup.template.name: '{CLOUD_APP}' setup.template.pattern: '{CLOUD_APP}-*'
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 1
output.elasticsearch:
hosts: ['https://es_ip:19200']
username: 'abc'
password: 'yoyo'
indices:
- index: 'auth-multi-%{+YYYY.MM.dd}'
when.contains:
source: '/var/log/auth.log'
- index: 'audit-multi-%{+YYYY.MM.dd}'
when.contains:
source: '/var/log/audit/audit.log'
- index: 'syslog-multi-%{+YYYY.MM.dd}'
when.contains:
source: '/var/log/syslog'

another configuration as below under /etc/filebeat1
filebeat.config.prospectors:
enabled: true
path: /etc/filebeat1/conf.d/*.yml
reload.enabled: true
reload.period: 5s

setup.template.name: 'elk'
setup.template.pattern: 'elk-*'
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 1
output.logstash:
when:
contains:
source: '/home/elk/elasticsearch_logs/Hotdata-node/logs/Demo.log'
hosts: ["xyz:5044"]

3. I also configured 2 service as filebeat.service and filebeat1.service and was able to successfully push the logs in elasticsearch.

My concerns:
a. Is it right method in case we want to push logs to multiple output?
b. what all factor i need to consider if i place above configurations in production. Will running multiple instances of filebeat can cause any resource impact on the server on long run?

[shaunak (Shaunak Kashyap)]

a. Yes, if you need different outputs, you'll need to run different Filebeat instances for these, one per unique output. This may change in the future (we may offer a single binary) but for now, running multiple instances is the right way.

b. I don't think you need to worry about resource impact but I would make sure that each Filebeat instance has its own, unique path.data directory. If you run each Filebeat instance with the export config subcommand it will output that instance's configuration in YML format. In that output look for the path.data key. You will want to make sure the value is different for each Filebeat instance. You can read about how to configure these paths here: https://www.elastic.co/guide/en/beats/filebeat/master/configuration-path.html.

[mathur7vidit (vidit mathur)]

hi shaunak,

yes i kept different path.data so that there should not be anything common across 2 filebeat instance on the same machine. also, i registered 2nd instance as filebeat1.service which is working as expected. So do you think we can plan this setup for prod environment. major concern is around Resource utilization CPU, mem on prod servers which should not impact application running on those servers in prod.