I'm trying to use filebeat in Docker to read log files from local machine and output to Kafka. But it doesn't work properly, doesn't output any message to Kafka. And I tested Kafka on my local machine, it works well itself.
On my local machine, the file and folder structure like this:
The content of Dockerfile like below:
FROM docker.elastic.co/beats/filebeat:7.4.2
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
USER filebeat
The Content of filebeat.yml like below:
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /Users/lachen/Documents/Log-Monitor/nifi-bootstrap-3.log
#----------------------------- Kafka output --------------------------------
output.kafka:
# 10.133.93.243 is the private ip address of my local machine
# even though I use localhost as ["localhost:9092"], and it doesn't work properly
hosts: ["10.133.93.243:9092"]
topic: pam-filebeat-test2
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
1. I build the filebeat image using below command:
docker image build -t filebeat:1.0.2 .
2. I run the filebeat container using below command
docker container run -u 0 --name filebeat-container filebeat:1.0.2
I also tried to run the container using command like below, it doesn't work as well:
docker container run -u 0 -e strict.perms=false --name filebeat-container filebeat:1.0.2
3. I check the topic in kafka using below command:
kafka-console-consumer --bootstrap-server localhost:9092 --topic pam-filebeat-test2 --from-beginning
4. add new contents to the logfiles
echo "1">>nifi-bootstrap-3.log
echo "2">>nifi-bootstrap-3.log
echo "3">>nifi-bootstrap-3.log
5. go back and check the kafka, it didn't receive any message from the filebeat.
