Filebeat-unable-to-send-logs-to-kafka

Elastic Stack Beats
1

File Beat is unable to send logs from a particular folder, This is the application logs folder.

Things that have been tried :

  • Created a new topic in kafka to retest the settings.
  • Checked for file permission for the folder and the file to send.
  • Updated to the filebeats to 6.7 from 5.5
  • changed from filebeat.prospector to filebeat.inputs

Running Configuration

filebeat.inputs:
- type: log
  paths:
    - /var/log/containers/*.log
fields_under_root: true
output.kafka:
  hosts: ["10.0.0.0:9092"]
  topic: "testtopic"
  codec.json:
    pretty: true

With this i am able to see all the logs in "testtopic"

Not running Configuration :

filebeat.inputs:
- type: log
  paths:
    - /app/log/server/*.log
fields_under_root: true
output.kafka:
  hosts: ["10.0.0.0:9092"]
  topic: "testtopic"
  codec.json:
    pretty: true

Expected Results : Logs from the path /app/log/server/*.log should be sent to Kafka

2

Looks like the only difference between the "running" and "not running" configurations is the path to the logs folder. Could you start up Filebeat with the "non running" configuration and post the first 50 or so Filebeat log lines here? Please be sure to redact any sensitive information before posting.

3

Hi @shaunak,
The issue have been resolved, The issue was that for the Volume Mounts for Kubernates.
I have to mount the paticular volume to allow filebeat to process the logs forward.

Thanks for the response.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.