One prospector stops

Elastic Stack Beats
1

Hello,

I have a filebeat config with two prospectors and one stops working after a couple days. The other one keeps on working normally.

Is there any way to debug why this is happening. I checked the log files and it just logs that there are no changes in the file.

After the restart it finds the new data.

Kind regards,

Uros

2

What version are you on? What OS?
Have you tried running with debug?
What differences are there between the two?

3

Hello,

OS is Win 2012 R2, filebeat is 1.2.3.

Yes, I have running it with debug, but there are no errors in the logs. It only logs no changes in file (Not harvesting, file didn't change:)

The prospectors are identical, expect for one exclude_lines directive in the prospector that is the problem.

Kind regards,

Uros

4

Are you fetching files from a mounted volume?

5

It´s a normal windows disk drive.

6

Can you share your config file? Does it work if you remove the exclude_lines directive?

7

Here is my config:

filebeat:
  prospectors:
    -
      paths:
        - E:/SoftwareAG/ARIS9/server/bin/work/work_businesspublisher_m/base/webapps/businesspublisher/log/monitoring/webAccess-*.log
      encoding: utf-8
      input_type: log
      exclude_lines: ["^#"]
      document_type: arislogweb
    -
      paths:
        - E:/SoftwareAG/ARIS9/server/bin/work/work_businesspublisher_m/base/webapps/businesspublisher/log/monitoring/modelAccess-*.log
      encoding: utf-8	  
      input_type: log
      document_type: arislogmodel
      close_older: 1h
  registry_file: "C:/ProgramData/filebeat/registry"

output:
  logstash:
      hosts: ["somehost:5044"]
    tls:
      certificate_authorities: ["C:/Program Files/filebeat/chain.cer"]
      certificate: "C:/Program Files/filebeat/cert.cer"
      certificate_key: "C:/Program Files/filebeat/cert.key"

logging:
   to_files: true
   files:
     path: C:/ProgramData/filebeat
     name: filebeat_debug
     rotateeverybytes: 10485760 # = 10MB
     keepfiles: 7
   selectors: ["*"]
   level: debug

I have just removed the closed_older from the top prospector. Just for testing. Next I will remove the exclude directive, probably tomorrow.

Kind regards,
Uros

8

did just skim over your config, but: Never use double quotes " for regular expressions, but use single quotes '. There are 5 different kind of string formats in YAML with different escaping rules.

9

Thanks. I changed the exclude_lines directive...or did you mean I should use single quotes everywhere in the config?

Kind regards,

Uros

10

it's up to you when you use single quotes. Advantage of single-quotes is, you don't have to deal with YAML based escaping rules. For example windows file paths can be written with backslash (e.g. copy'n paste path) if single quote is used.

11

Thanks. Didn't know that.

12

Hello,

could it be that it has something to do with the way the original app is writing the log file. It looks kinda strange, since there are changes in the log file, but the last access or last write date do not get changed. Could that have to do something with filebeat?

Kind regards,

Uros

13

@Uros_Meglic Yes, that is a common problem with shared drives. It is recommended to install filebeat on each edge server. There are some improvements in the 5.0.0-alpha releases related to this in case you are interested to try it out: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

14

This log file is not on a shared (network) drive or what do you mean exactly with shared drives?

Can I use the 5.0 filebeat with older logstash server?

Kind regards,

Uros

16

This topic was automatically closed after 21 days. New replies are no longer allowed.