# Logstash not ingesting when new file added to the directory **URL:** \ **Category:** Logstash\ **Created:** [July 22, 2016, 9:23pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206 "2016-07-22T21:23:36Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 22, 2016, 9:23pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/1 "2016-07-22T21:23:36Z")
I have a logstash configured to ingest S3 access logs , i can successfully ingest data to Elasticsearch when i restart the logstash without any issues. But when new files gets added to the directory from where the Logstash pull the data from, logstash does nothing. I am not sure what's going wrong. The permission and everything is correct. Below is my logstash configuration. `input { file { type => "s3-access-log-new" path => "/home/ubuntu/s3logs/logs/*" start_position => "beginning" } } filter { if [type] == "s3-access-log-new" { grok { match => { "message" => "%{S3_ACCESS_LOG}" } } date { locale => "en" match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"] } } } output { elasticsearch { host => ["elk-prod-02-data01"] port => "9200" protocol => "http" index => niraj-log-s3-new-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } }` What am i doing wrong? - Niraj ---
**Author:** ![jpcarey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpcarey/32/46668_2.png) [@jpcarey](https://discuss.elastic.co/u/jpcarey)\ **Post date:** [July 23, 2016, 4:10am UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/2 "2016-07-23T04:10:49Z")
Is there a reason you aren't using the s3 input plugin? ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 25, 2016, 4:31pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/3 "2016-07-25T16:31:35Z")
Is there a reason i should use it in replacement for file based logging? I mean i am not a logstash expert though but i haven't tried that. Will using the s3 plugin fix my problem? ---
**Author:** ![jpcarey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpcarey/32/46668_2.png) [@jpcarey](https://discuss.elastic.co/u/jpcarey)\ **Post date:** [July 25, 2016, 4:35pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/4 "2016-07-25T16:35:56Z")
Not necessarily. Just curious how you are syncing or pulling in new files to this directory. What version of logstash are you using? ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 25, 2016, 4:43pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/5 "2016-07-25T16:43:37Z")
So it goes like this. S3cmd tool used to sync files using a cron. The sync is continuous on an interval of 30 mins. LS Version:- 1.5.3 ES Version:- 1.5.1 ---
**Author:** ![jpcarey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpcarey/32/46668_2.png) [@jpcarey](https://discuss.elastic.co/u/jpcarey)\ **Post date:** [July 25, 2016, 5:41pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/6 "2016-07-25T17:41:25Z")
You should upgrade to at least LS 1.5.6 - the last release of the 1.5 series. It uses the ruby filewatch 0.6.7 library for the file input. filewatch 0.6.4 had a bug which lost file tracking info. ---
**Author:** ![jpcarey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpcarey/32/46668_2.png) [@jpcarey](https://discuss.elastic.co/u/jpcarey)\ **Post date:** [July 25, 2016, 5:52pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/7 "2016-07-25T17:52:34Z")
And I just looked, the 1.0.0 logstash-input-s3 plugin does not use filewatch library, rather it downloads and processes directly (and has an option to save a copy of the file). This should work around the issue that you are seeing, and provide the same level of functionality. ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 25, 2016, 5:54pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/8 "2016-07-25T17:54:24Z")
Let me try that and see if that works. I believe LS version is not tagged to a specific ES version, like other plugins do. ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 25, 2016, 7:03pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/9 "2016-07-25T19:03:53Z")
I just installed this version , but still the behavior seems to be the same. ---
**Author:** ![jpcarey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpcarey/32/46668_2.png) [@jpcarey](https://discuss.elastic.co/u/jpcarey)\ **Post date:** [July 25, 2016, 7:47pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/10 "2016-07-25T19:47:36Z")
What does `tree /home/ubuntu/s3logs/logs/` or `ls -la /home/ubuntu/s3logs/logs/` look like? ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 25, 2016, 8:21pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/11 "2016-07-25T20:21:45Z")
Something like below: -rwxrwxr-x 1 ubuntu ubuntu 381 Jul 25 19:43 2016-07-25-19-43-56-C970EE64265A8BC4 -rwxrwxr-x 1 ubuntu ubuntu 382 Jul 25 19:44 2016-07-25-19-44-18-651834568545399D -rwxrwxr-x 1 ubuntu ubuntu 387 Jul 25 19:46 2016-07-25-19-46-19-CB7B8C1AD2B33C72 -rwxrwxr-x 1 ubuntu ubuntu 306 Jul 25 19:46 2016-07-25-19-46-21-1B1792E2EBD80D00 -rwxrwxr-x 1 ubuntu ubuntu 612 Jul 25 19:47 2016-07-25-19-47-30-68807DA9BA824DD8 -rwxrwxr-x 1 ubuntu ubuntu 383 Jul 25 19:48 2016-07-25-19-48-12-5A8DADC21D9EF463 -rwxrwxr-x 1 ubuntu ubuntu 306 Jul 25 19:48 2016-07-25-19-48-17-BF00BFFFB90B48F1 -rwxrwxr-x 1 ubuntu ubuntu 880 Jul 25 19:48 2016-07-25-19-48-58-FB66AE7E7FA4D696 -rwxrwxr-x 1 ubuntu ubuntu 306 Jul 25 19:50 2016-07-25-19-50-39-B25B166C8EA87D8D -rwxrwxr-x 1 ubuntu ubuntu 382 Jul 25 19:52 2016-07-25-19-52-42-B38935F89AF1A485 -rwxrwxr-x 1 ubuntu ubuntu 1881 Jul 25 19:54 2016-07-25-19-54-06-3D24B32CB2E53D46 ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 26, 2016, 6:11pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/12 "2016-07-26T18:11:15Z")
@jpcarey, Is there a way to find out the the file input plugin i have currently is using the ruby filewatch library. I did update the logstash, but i believe the re-install didn't update the plugin. ---
**Author:** ![jpcarey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpcarey/32/46668_2.png) [@jpcarey](https://discuss.elastic.co/u/jpcarey)\ **Post date:** [July 26, 2016, 6:14pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/13 "2016-07-26T18:14:15Z")
`bin/plugin list --verbose | grep file` Should be `logstash-input-file (1.0.2)`. ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 26, 2016, 6:19pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/14 "2016-07-26T18:19:58Z")
Thanks @jpcarey. I am really scratching my head now 😄 The input plugin is fine as well. I am really not sure what is going wrong. ---
**Author:** ![jpcarey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpcarey/32/46668_2.png) [@jpcarey](https://discuss.elastic.co/u/jpcarey)\ **Post date:** [July 26, 2016, 7:01pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/15 "2016-07-26T19:01:33Z")
I would assume it has something to do with how the s3cmd downloads and creates the files. I would guess that it creates some temporary files during download. This would require some in depth troubleshooting to figure out what all is happening, and why logstash does not pick up (or potentially believe that it has already processed the file). You might try appending an extension for the finished file (if you can do this with s3cmd or control the s3 access log naming pattern). Then, configure logstash to look for `*.my_file_type`. It should then ignore any temporary files in the directory. Alternatively, try the s3 input. ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 28, 2016, 5:48pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/16 "2016-07-28T17:48:07Z")
I tested this out today on a different box with same setup as my EC2 instance. But on my local workstation it works perfectly fine and detects new files when added to the directory by s3cmd. I will try the s3 input and see if that takes care of the issue. ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [July 28, 2016, 6:00pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/17 "2016-07-28T18:00:16Z")
Any idea why i see the below error on s3 plugin . A plugin had an unrecoverable error. Will restart this plugin. Plugin: \"s3-access-log", bucket=\>"niraj-s3-log", prefix=\>"logs", region=\>"us-east-1", access\_key\_id=\>"xxxxxxxxxxxxxxxxxxxxxxxxx", secret\_access\_key=\>"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", debug=\>false, codec=\>\"UTF-8"\>, use\_ssl=\>true, delete=\>false, interval=\>60, temporary\_directory=\>"/tmp/logstash"\> Error: certificate verify failed {:level=\>:error} A plugin had an unrecoverable error. Will restart this plugin. Plugin: \"s3-access-log", bucket=\>"niraj-s3-log", prefix=\>"logs", region=\>"us-east-1", access\_key\_id=\>"xxxxxxxxxxxxxxxxxxxxxxxxx", secret\_access\_key=\>"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", debug=\>false, codec=\>\"UTF-8"\>, use\_ssl=\>true, delete=\>false, interval=\>60, temporary\_directory=\>"/tmp/logstash"\> Error: certificate verify failed {:level=\>:error} ^CSIGINT received. Shutting down the pipeline. {:level=\>:warn} Logstash shutdown completed ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:46am UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206/18 "2017-07-06T04:46:06Z")