# Mulitple Filebeat Instances
**URL:** https://discuss.elastic.co/t/mulitple-filebeat-instances/330792
**Category:** Beats
**Tags:** beats-module, filebeat
**Created:** [April 26, 2023, 6:30am UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792 "2023-04-26T06:30:59Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@BoKu](https://discuss.elastic.co/u/BoKu)
#### Post date: [April 26, 2023, 6:31am UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/1 "2023-04-26T06:31:00Z")
Hello,
i did setup two filebeat instances on a linux server. One for Syslog and the PANW-Module and the other for the F5-Module.
The Syslog/PANW Filebeat was the first one, i did change the index to a different one, but it automatically create a datastream.
To setup the second instance i created an additional systemd entry and copied the original etc-Directory and configured the filebeat.yml (i just change the indexname and disbaled Syslog) and activated f5. Upon starting the second instance, it creates an index without ILM and does not create an datastream.
Any Idea why the second instance does not create an datastream?
Thanks in advance!
Regards Boris
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 2, 2023, 3:16am UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/2 "2023-05-02T03:16:14Z")
Is there a reason you are running an instance per module, a single instance should handle this.
---
### Author:  [@BoKu](https://discuss.elastic.co/u/BoKu)
#### Post date: [May 2, 2023, 5:12am UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/3 "2023-05-02T05:12:12Z")
Hello Mark,
yes there is. Since the update to 8.7.0 the Filebeat f5-Modul sends less data than before the update (Verified with parallel sending the f5-data to logstash).
I just wanted to seperate the f5-module to have own Logs and an own Index just for the data coming from the f5, to further investigate this issue.
Kind regards
Boris
---
### Author:  [@willemdh](https://discuss.elastic.co/u/willemdh)
#### Post date: [May 2, 2023, 7:39pm UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/4 "2023-05-02T19:39:08Z")
@warkolm There are so many reasons there is a need for multiple agents on a host. One example, which is applicable to Elastic ecosystem itself, a customer typically needs to forward the Elasticsearch / Logstash / Kibana logs and metrics to a separate monitoring cluster. This is not possible in general, as there is already a set of agents running on this node to index system logs and metrics..
@BoKu We did some custom scripting with Ansible to achieve this, but it has been a pain tbh..
Elastic should really support multiple outputs or provide a supported way to install and mange multiple identical agents on a system. Even Elastic Agent is very limited in that way, I'm not even sure it's possible with Agent.. (eg sending system logs to production cluster and elastic logs and metrics to a separate monitoring cluster).
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 2, 2023, 9:55pm UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/5 "2023-05-02T21:55:53Z")
@willemdh
I think the team is looking at defining an output per integration perhaps you should add your thought to this public issue...
I also see it listed on our internal tracking issues.
Your feedback is timely as I think they are gather feedback right now... this is just a discus forum your voice will be more likely heard in an issue.
BTW This is for elastic-agent, I would not expect this to get backported to filebeat "solo"
>
>
> There are many legitimate reasons why an operator may need/want to send data fro…m integrations to different outputs within a policy. Some may even need to send datastream to different outputs. Currently we only allow an output to be defined on a per policy basis. In order to support this request the per policy output definition needs to be over-written by the output defined in the integration. Our config should support this already.
>
> \*\*Use Cases:\*\*
>
> 1) As an operator, I need my security logs from an agent to be sent to one logstash where as informational logs to be sent to another logstash instance.
>
> 2) We operate multiple beats on a given system and would like to migrate to using Elastic Agent. For historical and operational reasons these beats are writing data to distinct outputs. Once we migrate over to using Agent, we would like to keep the upstream pipeline intact.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 2, 2023, 10:24pm UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/6 "2023-05-02T22:24:40Z")
My 2nd thought on this, when I got it working (it has been a while), was to 1st install 1 completely separate tar.gz install for the 2 beats get it all working correctly.
Then I worked on getting them into systemctl etc...
I suspect you may be stepping on the data path... maybe ... maybe not.
---
### Author:  [@BoKu](https://discuss.elastic.co/u/BoKu)
#### Post date: [May 3, 2023, 5:41am UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/7 "2023-05-03T05:41:01Z")
Hello and thanks for your replys,
but i had another initial question. The second instance is runing and working BUT it creates an index without ILM instead of a datastream.
I need to have an individual (something like: filebeat-f5-%{[agent.version]}) index with ILM or a working datastream.
Regards Boris
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 3, 2023, 1:35pm UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/8 "2023-05-03T13:35:53Z")
You will need to share your entire filebeat.yml and whatever module your are using otherwise we are just guessing.
Please show the exact current configuration and results
---
### Author:  [@BoKu](https://discuss.elastic.co/u/BoKu)
#### Post date: [May 16, 2023, 6:13am UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/9 "2023-05-16T06:13:53Z")
Here is the filebeat.yml
```auto
###################### Filebeat Configuration Example #########################
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
# Change to true to enable this input configuration.
enabled: false
## Paths that should be crawled and fetched. Glob based paths.
#paths:
# - /var/log/httpd/*log
# #- c:\programdata\elasticsearch\logs\*
#tags: ["httpd_rh_log"]
- type: syslog
enabled: false
protocol.udp.host: "0.0.0.0:514"
tags: ["syslog", "udp"]
- type: syslog
enabled: false
protocol.tcp.host: "0.0.0.0:514"
tags: ["syslog", "tcp"]
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
#================================ General =====================================
tags: ["f5"]
# Optional fields that you can specify to add additional information to the
# output.
fields:
env: prod
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "https://localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id: ""
#============================= Elastic Cloud ==================================
#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["5_HOSTS:9200]
# Optional protocol and basic auth credentials.
protocol: "https"
username: "beatwriter"
password: "${PWD}"
ssl.certificate_authorities: ["/path/to/CA_cert"]
#index: "filebeat-f5-%{[agent.version]}"
index: "filebeat-f5-%{[agent.version]}"
#setup.template.name: "filebeat"
#setup.template.pattern: "filebeat"
#----------------------------- Logstash output --------------------------------
#================================ Setup ==========================================
setup.template.name: "filebeat-f5-%{[beat.version]}"
setup.template.pattern: "filebeat-f5-%{[beat.version]}-*"
setup.ilm.enabled: true
setup.ilm.rollover_alias: "filebeat-f5-%{[agent.version]}"
setup.ilm.policy_name: "filebeat"
#================================ Processors =====================================
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
#================================ Logging =====================================
logging:
level: warning
to_files: true
to_syslog: false
json: true
files:
path: '/var/log/filebeat-f5'
name: 'filebeat'
keepfiles: '3'
permissions: '0644'
#============================== X-Pack Monitoring ===============================
#================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
```
and the module-config f5.yml
```auto
# Module: f5
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.16/filebeat-module-f5.html
- module: f5
bigipapm:
enabled: true
# Set which input to use between udp (default), tcp or file.
var.input: udp
var.syslog_host: 0.0.0.0
var.syslog_port: 9504
# Set paths for the log files when file input is used.
# var.paths:
# Toggle output of non-ECS fields (default true).
var.rsa_fields: true
var.keep_raw_fields: true
# Set custom timezone offset.
# "local" (default) for system timezone.
# "+02:00" for GMT+02:00
# var.tz_offset: local
bigipafm:
enabled: true
# Set which input to use between udp (default), tcp or file.
# var.input: udp
# var.syslog_host: localhost
# var.syslog_port: 9528
# Set paths for the log files when file input is used.
# var.paths:
# Toggle output of non-ECS fields (default true).
# var.rsa_fields: true
# Set custom timezone offset.
# "local" (default) for system timezone.
# "+02:00" for GMT+02:00
# var.tz_offset: local
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 16, 2023, 12:44pm UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/10 "2023-05-16T12:44:24Z")
What version are you on?
And what resulting index/data stream is created with this configuration?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 13, 2023, 2:44pm UTC](https://discuss.elastic.co/t/mulitple-filebeat-instances/330792/11 "2023-06-13T14:44:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.