Filebeats not harvesting new file

[NickRoper]

Hi, I'm new to the ELK stack so apologies if this is a stupid question.

I've downloaded and installed the following:

• elasticsearch-6.5.4
• kibana-6.5.4
• logstash-6.5.4
• filebeats-6.54

I have everything installed and have successfully indexed the example apache logs into elasticsearch & kibana by using filebeats. In the

filebeat.inputs:
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/nick.roper/elasticsearch/data/apache-test/*

Initially the directory just contained an example 'logs' file and this was processed fine.

I then created a new file in the same directory and expected it to be found and processed by filebeats, but it doesn't appear in Kibana, even after several minutes. Am I missing something?

Thanks

[pierhugues (Pier-Hugues Pellerin)]

@NickRoper Is there any error in the Filebeat log? Does Filebeat has the permission to read that file?

[NickRoper]

Hi @pierhugues, thanks for getting back.

I can't see anything in the logs, but I did try restarting Filebeat and then it started harvesting from the new file and picking up any new entries OK.

I originally added the new file after filebeat had been started the first time, so maybe it finds any files that match the pattern when it starts up and then only watches those - could that be the case?

[pierhugues (Pier-Hugues Pellerin)]

That should not be the case, we periodically scan the disk for new files, if you start filebeat with -e -d "*" we should a more information when it discover files or why its not considering the file to harvesting.

[NickRoper]

Thanks @pierhugues - useful to know about those flags and I could see it scanning the files. It seems to be working OK now, so I'm not sure what the issue was previously, but many thanks for your help.