Hi, I'm new to the ELK stack so apologies if this is a stupid question.
I've downloaded and installed the following:
elasticsearch-6.5.4
kibana-6.5.4
logstash-6.5.4
filebeats-6.54
I have everything installed and have successfully indexed the example apache logs into elasticsearch & kibana by using filebeats. In the
filebeat.inputs:
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /Users/nick.roper/elasticsearch/data/apache-test/*
Initially the directory just contained an example 'logs' file and this was processed fine.
I then created a new file in the same directory and expected it to be found and processed by filebeats, but it doesn't appear in Kibana, even after several minutes. Am I missing something?
I can't see anything in the logs, but I did try restarting Filebeat and then it started harvesting from the new file and picking up any new entries OK.
I originally added the new file after filebeat had been started the first time, so maybe it finds any files that match the pattern when it starts up and then only watches those - could that be the case?
That should not be the case, we periodically scan the disk for new files, if you start filebeat with -e -d "*" we should a more information when it discover files or why its not considering the file to harvesting.
Thanks @pierhugues - useful to know about those flags and I could see it scanning the files. It seems to be working OK now, so I'm not sure what the issue was previously, but many thanks for your help.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.