Multiple filebeat_<host>.yml files from a central log sever

[penguingeek]

I have a central log server that i want to forward logs to logstash from. The issue I am having is that since I am forwarding logs to the central server, there are no logs stored locally for filebeat to pull from. Now i am trying to separate those logs from the central server to ship to logstash, to then be forwarded on to elasticsearch (into host-based indexes). I was using "filebeat.config.prospector"
but from what I can tell from the documentation, I can only have separate prospectors, not multiple full conf.d/.yml files.

I am using:
64bit CentOS 7
Filebeat 6.2.2

filebeat.yml:

filebeat.config.prospectors:
enabled: true
path: /etc/filebeat/conf.d/*.yml

conf.d/.yml:

filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log//*.log
exclude_files: [".gz$"]

name:
tags: [""]
logstash.registry_file: /var/lib/filebeat/registry

setup.dashboards.enabled: false
setup.kibana:
host: ":5601"

output.logstash:
hosts: [":10514"]

logging.level: info

[penguingeek]

I managed to work around this issue by changing the "name:" value into a tag and moving the tags under "logstash.prospecters:" in all the .yml files under conf.d/

This isn't ideal, as I was hoping to do some preprocessing before logstash, but this at least lets me move forward.

[ruflin]

I'm glad you found a solution. What is the "preprocessing" step you can't do?

[penguingeek]

Hey ruflin,

Basically logs from a particular set of hosts are being forwarded to a central server and no logs are being kept local. So I am trying to run filebeat from the central log server and maintain the "from_host" as the original sending host, not the central server.

I was hoping i could do this with individual yml files, but after looking through all the documentation, it seems like I can only do this with tags under the prospector section. That's a lot of config management to have separate config files just for hostname tags.

[ruflin]

As you probably know filebeat is expected to run on the edge. You state above that there are no logs on the edge machine because they are forwarded to a central server. So at some point the logs seem to be there, why not directly use filebeat on the edge servers. This would simplify your setup I would hope.