Filebeat stops processing files after 1 or 2 days issue


#1

Hi,

I have Filebeat configured to fetch data from log files in a folder. I have log files, which is sliced to 10 mb log fies in my linux box, and so I configured filebeat to scan the entire folder. There will be approx 10k entries per day. I am forwarding filebeat entries to kafka, and from kafka to logstash, and finally to ES.

FB --> KFK --> LS --> ES

but the thing is that, sometimes after 1 day after i start logstash, the process will be running, but there will be no indexes in elasticsearch for that day. Or sometimes, there will be 5 or 6 entries for that day. Once i terminate the filebeat process and restarts it, then the entry gets populated as normal and everything works fine.

Below is my filebeat configuration.

filebeat.prospectors:
- input_type: log
  paths:
    - /logfolder/logfile*
  exclude_files: [".lck"]
  fields:
    logtype: LOGFILE
  document_type: dummylog
  scan_frequency: 1m
  close_inactive: 10s
  multiline.pattern: '^<[A-Za-z_]{3} [[:digit:]]{2}, [[:digit:]]{4} ([[:digit:]]{1}|[[:digit:]]{2}):[[:digit:]]{2}:[[:digit:]]{2}:([[:digit:]]{1}|[[:digit:]]{2}|[[:digit:]]{3}) [A-Z]{2}>'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 5000
filebeat.registry_file: .regfile
output.kafka:
  enabled: true
  hosts: ["192.168.1.1:9092"]
  topic: filebeatlogs
  worker: 1
  max_retries: 2
output.file:
  enabled: true
  path: "/outputfolder/logdata"
  filename: filebeatlogs

How can I fix this?


(ruflin) #2

Which version of Filebeat are you using? Which version of Logstash?


#3

filebeat-5.0.0-alpha5-linux-x86_64
logstash-2.3.4


(ruflin) #4
  • Could you update to the 5.0 GA release?
  • Can you share some log files with at least INFO level from filebeat?

#5

lemme update to 5.0 GA and reply. Thanks.


(system) #6

This topic was automatically closed after 21 days. New replies are no longer allowed.