# Problem with large files & output

**URL:** <https://discuss.elastic.co/t/problem-with-large-files-output/49378>\
**Category:** Logstash\
**Created:** [May 6, 2016, 8:10am UTC](https://discuss.elastic.co/t/problem-with-large-files-output/49378 "2016-05-06T08:10:28Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\
**Post date:** [May 6, 2016, 8:10am UTC](https://discuss.elastic.co/t/problem-with-large-files-output/49378/1 "2016-05-06T08:10:28Z")

</div>

I have a case that works fine on logstash 1.5.x but breaks in 2.2, 2.3, and 5.0 (I haven't tried 2.0 or 2.1). I have hourly syslog files that are approximately 3 gig each. I'm trying to backfill logs going back to January. Here's my config on my shipper:

input {  
file {  
path =\> "/raid0/logs/201601\*.log"  
start\_position =\> "beginning"  
}  
file {  
path =\> "/raid0/logs/201602\*.log"  
start\_position =\> "beginning"  
}  
file {  
path =\> "/raid0/logs/201603\*.log"  
start\_position =\> "beginning"  
}  
file {  
path =\> "/raid0/logs/201604\*.log"  
start\_position =\> "beginning"  
}  
file {  
path =\> "/raid0/logs/201605\*.log"  
start\_position =\> "beginning"  
}  
}

output {  
rabbitmq {  
durable =\> "false"  
exchange =\> "logstash.crunch"  
exchange\_type =\> "direct"  
host =\> "XXXXXXXX"  
password =\> "XXXXXX"  
user =\> "lXXXXXX"  
}  
}

Rabbit sees 0 traffic. I've also tried with stdout and file outputs and ended up with nothing. Logstash just sits practically idle.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [May 6, 2016, 2:28pm UTC](https://discuss.elastic.co/t/problem-with-large-files-output/49378/2 "2016-05-06T14:28:10Z")

</div>

@Janet This is really weird it should just be able to read everything, Any errors in the logs? You don't have any filters? If you run it with `--debug` do you get more details? Warning theses log can get a bit more noisy.

---

<div class="post-metadata">

**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\
**Post date:** [May 7, 2016, 1:54am UTC](https://discuss.elastic.co/t/problem-with-large-files-output/49378/3 "2016-05-07T01:54:52Z")

</div>

I simplified the config as much as I could to narrow down the issue. The logs showed nothing. I’ll try —debug and let you know what happens.

---

<div class="post-metadata">

**Author:** ![MrLee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrlee/32/9429_2.png) [@MrLee](https://discuss.elastic.co/u/MrLee)\
**Post date:** [May 7, 2016, 2:36am UTC](https://discuss.elastic.co/t/problem-with-large-files-output/49378/4 "2016-05-07T02:36:15Z")

</div>

The plugin "logstash-input-file" has an argument called "ignore\_older", by default, it will ignore the old file which was last modified before 24 hours. You can try to set it bigger to see whether it works well or not.

```auto
ignore_older

Value type is number
Default value is 86400
When the file input discovers a file that was last modified before the specified timespan in seconds, the file is ignored. After it’s discovery, if an ignored file is modified it is no longer ignored and any new data is read. The default is 24 hours.

```

---

<div class="post-metadata">

**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\
**Post date:** [May 7, 2016, 4:15am UTC](https://discuss.elastic.co/t/problem-with-large-files-output/49378/5 "2016-05-07T04:15:40Z")

</div>

I ran a script to touch all the files and update their timestamps - now they work fine. Thanks for the 86400 tip!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:58am UTC](https://discuss.elastic.co/t/problem-with-large-files-output/49378/6 "2017-07-06T04:58:56Z")

</div>

Closed on July 6, 2017, 4:58am UTC