Filebeat not polling the directory

Elastic Stack Beats
1

if I run the command:

filebeat.exe -c filebeat.yml -e

then any logs that are dropped into the directory I specify will be harvested and sent to kafka, as expected. But I do not want a command window up and running. I thought that if you just start the Filebeat service it would poll the directory and anytime a new log was dropped, it would harvest and send. That is not the case, I start the service and if any new logs are there when it is started it will harvest, but any logs dropped after, it does nothing.

How can I get the service to continuously poll the path set in my prospector configuration?

Thanks,

2

Which user are you running Filebeat as when you run it as a service? Does that user have sufficient permissions to access the files? What's in the Filebeat logfile? You'll probably want to have verbose logging enabled.

3

so I was running this as the local service account, since then I have updated the service account to run as my user, which is a local admin. I still get the same issue, funny things is, the logs see the new file but it says

2016-06-02T16:29:28+01:00 DBG  Check file for harvesting: E:\Export\NUIX_LOGS\consilio_application_log_47_06-02-2016_1019.log
2016-06-02T16:29:28+01:00 DBG  Update existing file for harvesting: E:\Export\NUIX_LOGS\consilio_application_log_47_06-02-2016_1019.log
2016-06-02T16:29:28+01:00 DBG  Not harvesting, file didn't change: E:\Export\NUIX_LOGS\consilio_application_log_47_06-02-2016_1019.log

but if I stop the service and start it back up, with some new files in the dir, it will pick them up and publish them..BUT it doesn't get the files that were previously dropped and did not get published. Still, if I run the command "filebeat.exe -c filebeat.yml -e" it will get what was not published and send it over..

4

@kalapakim Please do not use 2 threads for the same conversation: Filebeat not polling the directory If you comment on others related to the same topic, please reference the other conversation.

5

Yea, I meant to reference it, my bad...

6

Here is my config file...
##################$$$###### Filebeat Configuration ############################

# This file is a full configuration example documenting all non-deprecated
# options in comments. For a shorter configuration example, that contains only
# the most common options, please see filebeat.yml in the same directory.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat prospectors =============================

# List of prospectors to fetch data.
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

# Type of the files. Based on this the way the file is read is decided.
# The different types cannot be mixed in one prospector
#
# Possible options are:
# * log: Reads every line of the log file (default)
# * stdin: Reads the standard in

#------------------------------ Log prospector --------------------------------
- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  # To fetch all ".log" files from a specific level of subdirectories
  # /var/log/*/*.log can be used.
  # For each file found under this path, a harvester is started.
  # Make sure not file is defined twice as this can lead to unexpected behaviour.
  paths:
    - E:\Export\NUIX_LOGS\*.log

  # Type to be published in the 'type' field. For Elasticsearch output,
  # the type defines the document type these entries should be stored
  # in. Default: log
  document_type: logging_test 

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#------------------------------- Kafka output ---------------------------------
output.kafka:
  # The list of Kafka broker addresses from where to fetch the cluster metadata.
  # The cluster metadata contain the actual Kafka brokers events are published
  # to.
  hosts: ["MTPVPSESLG02:9092"]

  # The Kafka topic used for produced events. If use_type is set to true, the
  # topic will not be used.
  topic: "logging_test"
  channel: "logging_test"

  # Set Kafka topic by event type. If use_type is false, the topic option must
  # be configured. The default is false.
  #use_type: false

  # The number of concurrent load-balanced Kafka output workers.
  worker: 1
  
#----------------------------- Console output ---------------------------------
output.console:
  # Pretty print json event
  pretty: true

#================================= Paths ======================================

# The home path for the filebeat installation. This is the default base path
# for all other path settings and for miscellaneous files that come with the
# distribution (for example, the sample dashboards).
# If not set by a CLI flag or in the configuration file, the default for the
# home path is the location of the binary.
path.home: C:\nuixLogging\filebeat

# The configuration path for the filebeat installation. This is the default
# base path for configuration files, including the main YAML configuration file
# and the Elasticsearch template file. If not set by a CLI flag or in the
# configuration file, the default for the configuration path is the home path.
path.config: C:\nuixLogging\filebeat

#================================ Logging =====================================
# There are three options for the log output: syslog, file, stderr.
# Under Windows systems, the log files are per default sent to the file output,
# under all other system per default to syslog.

logging.to_files: true
logging.to_syslog: false
logging.level: debug
logging.files:
  path: C:\nuixLogging\filebeat\logs
  name: mybeat.log
  keep: 3
  rotateeverybytes: 10485760 # = 10MB
7

Could you share your full debug log output? It would be good to compare the output when you run it as service and when you run it manually. I assume you use the exact same config files?

8

I just added a bunch of logs and here is the log output, with the service running:
by the way, this is with FB 1.2, so this is happening with the stable version and the alpha 3 version, so I must have something configured incorrectly

2016-06-08T17:49:21+01:00 DBG  scan path E:\Export\NUIX_LOGS_fb123\*.log
2016-06-08T17:49:21+01:00 DBG  Check file for harvesting: E:\Export\NUIX_LOGS_fb123\consilio_application_log_62_06-03-2016_1054.log
2016-06-08T17:49:21+01:00 DBG  Same file as before found. Fetch the state.
2016-06-08T17:49:21+01:00 DBG  Check file for harvesting: E:\Export\NUIX_LOGS_fb123\consilio_application_log_63_06-03-2016_1054.log
2016-06-08T17:49:21+01:00 DBG  Same file as before found. Fetch the state.
2016-06-08T17:49:21+01:00 DBG  Check file for harvesting: E:\Export\NUIX_LOGS_fb123\consilio_application_log_64_06-03-2016_1054.log
2016-06-08T17:49:21+01:00 DBG  Same file as before found. Fetch the state.
2016-06-08T17:49:21+01:00 DBG  Check file for harvesting: E:\Export\NUIX_LOGS_fb123\consilio_application_log_65_06-03-2016_1054.log
2016-06-08T17:49:21+01:00 DBG  Same file as before found. Fetch the state.
2016-06-08T17:49:21+01:00 DBG  Check file for harvesting: E:\Export\NUIX_LOGS_fb123\consilio_application_log_66_06-03-2016_1503.log
2016-06-08T17:49:21+01:00 DBG  Same file as before found. Fetch the state.
2016-06-08T17:49:21+01:00 DBG  Check file for harvesting: E:\Export\NUIX_LOGS_fb123\consilio_application_log_67_06-03-2016_1503.log
2016-06-08T17:49:21+01:00 DBG  Same file as before found. Fetch the state.
2016-06-08T17:49:21+01:00 DBG  Check file for harvesting: E:\Export\NUIX_LOGS_fb123\consilio_application_log_68_06-03-2016_1503.log
2016-06-08T17:49:21+01:00 DBG  Same file as before found. Fetch the state.
2016-06-08T17:49:21+01:00 DBG  Check file for harvesting:

NOTE: After I stopped the service and kicked off the command, it harvested everything...

9

The above logs show, that it had harvested the same file before. I assume you have somewhere a registry file with the state inside. In case you want to harvest from scratch, please remove this file.

10

That's the problem, it says it has harvested it before, but it has not, so am I supposed to delete that registry file everytime? I'm literally dropping a new log, with a different filename, that includes datetime, so that the file is different and inside the file is logs that also has datetime. So the file is different. I till drop and then it will give these logs...never harvests the new files...

11

Do you delete an old file before you generate the new one? That sounds like a inode reuse issue.

12

This topic was automatically closed after 21 days. New replies are no longer allowed.