Multiple filebeat_<host>.yml files from a central log sever

I have a central log server that i want to forward logs to logstash from. The issue I am having is that since I am forwarding logs to the central server, there are no logs stored locally for filebeat to pull from. Now i am trying to separate those logs from the central server to ship to logstash, to then be forwarded on to elasticsearch (into host-based indexes). I was using "filebeat.config.prospector"
but from what I can tell from the documentation, I can only have separate prospectors, not multiple full conf.d/.yml files.

I am using:
64bit CentOS 7
Filebeat 6.2.2


enabled: true
path: /etc/filebeat/conf.d/*.yml


- type: log
enabled: true
- /var/log//*.log
exclude_files: [".gz$"]

tags: [""]
logstash.registry_file: /var/lib/filebeat/registry

setup.dashboards.enabled: false
host: ":5601"

hosts: [":10514"]

logging.level: info

I managed to work around this issue by changing the "name:" value into a tag and moving the tags under "logstash.prospecters:" in all the .yml files under conf.d/

This isn't ideal, as I was hoping to do some preprocessing before logstash, but this at least lets me move forward.

I'm glad you found a solution. What is the "preprocessing" step you can't do?

Hey ruflin,

Basically logs from a particular set of hosts are being forwarded to a central server and no logs are being kept local. So I am trying to run filebeat from the central log server and maintain the "from_host" as the original sending host, not the central server.

I was hoping i could do this with individual yml files, but after looking through all the documentation, it seems like I can only do this with tags under the prospector section. That's a lot of config management to have separate config files just for hostname tags.

As you probably know filebeat is expected to run on the edge. You state above that there are no logs on the edge machine because they are forwarded to a central server. So at some point the logs seem to be there, why not directly use filebeat on the edge servers. This would simplify your setup I would hope.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.