I am running a RHEL VM with Filebeats installed and three Podman containers running Kibana, Elasticsearch, and Logstash. Almost everything works, except for sending files from Filebeats to my Logstash container's pipeline.
I have tried and ensured the following:
- I verified my host machine's port 5044 is listening via ss -tulpn and that it's firewall is permitting traffic on the port. I can make a successful telnet connection to it. I've also ran Filebeat test output to get:
Logstash: 10.0.2.16:5044...
connection...
parse host... OK
dns lookup... OK
addresses: 10.0.2.16
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
- The Logstash Podman container was correctly mapped to the 5044 port upon execution. Verified by:
elastic@elk-server$ podman port ls01
5044/tcp -> 0.0.0.0:5044
9600/tcp -> 0.0.0.0:9600
- I've tested a Filebeat config which outputs to the host machine's console and it works. I've also tested Logstash and successfully ran pipelines which ingest a log file from a directory from within the container, Logstash can output both to stdout and Elasticsearch fine.
- I've continuously cleared both the Filebeat /var/lib/filebeat/registry/filebeat registry directory as well as my data/sincebd_ directory in Logstash
- I've also tried replacing my host machines IP (10.0.2.16) with localhost and 127.0.0.1 in my filebeat.yml file (shouldn't make a difference)
- I've put SELinux in permissive mode
- I don't have any TLS security configured between Filebeats and Logstash.
The contents of my filebeat.yml is:
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.inputs:
- type: filestream
enabled: true
id: FreeIPA-audit-log
paths:
- /home/elastic/audit.test.txt
parsers:
- multiline:
type: pattern
pattern: '^\s*$'
negate: true
match: after
output.logstash:
hosts: ["10.0.2.16:5044"]
And, the contents of my logstash.conf pipeline is:
input {
beats {
port => 5044
}
}
output {
file {
path => "/usr/share/logstash/output.log"
codec => line { format => "%{message}" }
}
}
I have also attempted to run a Logstash pipeline via command-line and output to stdout instead of a file, which also didn't work. I do not know what else to try at this point. Thank you for the help.