Filebeat automatically stops

Elastic Stack Beats
1

Hi,
I am using 6.2.3 version filebeat. Issue is my filebeat service gets automatically down very frequently. But when ever i made it up it will crawl properly and log data into elasticsearch properly for certain duration. But unexpectedly at some point of time(may be 1 or 2 hours time) filebeat service will get down. In logs also there is no exception stack trace. But there are other filebeat services which are running with no down time in the same machine. Is there any possibility where the filebeat gets stopped automatically?

Below is my filebeat configuration.

filebeat.prospectors:
- paths:
    - "/opt/logs/env1/env11/*/*.log"
    - "/opt/logs/env1/env12/*/*.log"
    - "/opt/logs/env1/env110/*/*.log"
    - "/opt/logs/env1/env111/*/*.log"
    - "/opt/logs/env1/env112/*/*.log"
    - "/opt/logs/env1/env113/*/*.log"
    - "/opt/logs/env1/env114/*/*.log"
    - "/opt/logs/env1/env115/*/*.log"
    - "/opt/logs/env1/env101/*/*.log"
    - "/opt/logs/env1/env102/*/*.log"
    - "/opt/logs/env1/env103/*/*.log"
    - "/opt/logs/env1/env104/*/*.log"
    - "/opt/logs/env1/env134/*/*.log"
    - "/opt/logs/env1/env135/*/*.log"
    - "/opt/logs/env1/env136/*/*.log"
    - "/opt/logs/env1/env138/*/*.log"
    - "/opt/logs/env1/env148/*/*.log"
  fields_under_root: true
  fields:
    df_env: env1
    df_portal: portal1
    df_application: service
  ignore_older: 2m
  close_inactive: 10s
  close_removed: true
  close_renamed: true
  clean_removed: true
  clean_renamed: true
  scan_frequency: 5s
  multiline.pattern: '^([[:alpha:]]{3,8}[[:space:]]{1,}\|[[:space:]]{1,}[[:digit:]]{1,4}-[[:digit:]]{1,2}-[[:digit:]]{1,4}[[:space:]]{1,})'
  multiline.negate: true
  multiline.match: after
  #tail_files: true
  #force_close_files: true

- paths:
    - "/opt/logs/env1/env116/*/*.log"
    - "/opt/logs/env1/env119/*/*.log"
    - "/opt/logs/env1/env117/*/*.log"
    - "/opt/logs/env1/env118/*/*.log"
  fields_under_root: true
  fields:
    df_env: env1
    df_portal: portal1
    df_application: service2
  ignore_older: 2m
  close_inactive: 10s
  close_removed: true
  close_renamed: true
  clean_removed: true
  clean_renamed: true
  scan_frequency: 5s
  multiline.pattern: '^([[:alpha:]]{3,8}[[:space:]]{1,}\|[[:space:]]{1,}[[:digit:]]{1,4}-[[:digit:]]{1,2}-[[:digit:]]{1,4}[[:space:]]{1,})'
  multiline.negate: true
  multiline.match: after
  #tail_files: true
  #force_close_files: true

filebeat.registry_file: /opt/filebeat/data/env1/portal1/env1_services

############################# Output ##########################################
output.kafka:
  hosts: ["env101.corp.xxx.com:9092","env102.corp.xxx.com:9092","env103.corp.xxx.com:9092"]
  topic: '%{[df_env]}_%{[df_portal]}_%{[df_application]}'
  worker: 10
  partition.round_robin:
    reachable_only: false

############################# Logging #########################################
logging.to_syslog: false
logging.selectors: ["beat"]
logging.level: info
logging.to_files: true
logging.files:
  path: /opt/filebeat/logs/env1/portal
  name: env1_portal.log
  rotateeverybytes: 10485760 # = 10MB
  keepfiles: 2

Thanks in advance

2

Just throwing an idea ~ depending on your OS's behavior, try monitoring your system resources (memory / cpu usage) and see if it's going smoothly or if it's being overloaded

3

Hi,

Have you checked your messages log ?
Path: /var/log/messages

Please check if you found something related to your config file issue.

Regards,

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.