# Logstash is not reading the log file(that's being updated automatically) continuously

**URL:** <https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686>\
**Category:** Logstash\
**Created:** [July 11, 2017, 3:45pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686 "2017-07-11T15:45:48Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [July 11, 2017, 3:45pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/1 "2017-07-11T15:45:48Z")

</div>

Hi, I have a log file that will be modified every 6 hours by a script. Problem is Logstash is not reading the file when the file is updated automatically. But when I update the file manually logstash is picking the data. I don't know the issue. Please help me on this issue. Here is my config file,

```
 input{
	file{
		path=>"C:/Path/Test.log"
		start_position=> "beginning"
		codec=> json
		sincedb_path => "NUL"
		ignore_older => 0
	}
}
output{
	stdout{
		codec=>rubydebug
		}
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 12, 2017, 7:09pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/2 "2017-07-12T19:09:45Z")

</div>

Exactly how is the file modified by the script?

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [July 12, 2017, 7:12pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/3 "2017-07-12T19:12:28Z")

</div>

It cleans the old data and writes new data every time and that is what I did manually and got it to work.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 12, 2017, 7:14pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/4 "2017-07-12T19:14:28Z")

</div>

That doesn't answer my question. Does the script just overwrite the old file? Is the file truncated? Before or after the file is truncated? These details matter.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [July 12, 2017, 7:16pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/5 "2017-07-12T19:16:18Z")

</div>

Hi Magnus, It just overwrites. File name remains same. What changes is just the data.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 12, 2017, 7:19pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/6 "2017-07-12T19:19:48Z")

</div>

Logstash doesn't discover such changes that happen behind its back. The reason it works for you is that your text editor (that I presume you used when you modified the file) actually creates a new file and renames it in place, replacing the original file. _That_ is picked up by Logstash. The reason it's doing so is because it's typically a safer way of updating files that your script should consider employing.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [July 12, 2017, 7:27pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/7 "2017-07-12T19:27:54Z")

</div>

Magnus, I am little bit confused here. Are you saying that there is no way logstash can support this scenario? What would be the alternative? Thanks for your help.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 12, 2017, 7:33pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/8 "2017-07-12T19:33:16Z")

</div>

The file input rereads a file if

- its inode number changes, or
- if the file shrinks (indicating that it was rotated via copy/truncate).

The file input does not support other means of detecting changes in a file.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [July 12, 2017, 8:08pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/9 "2017-07-12T20:08:54Z")

</div>

if the file shrinks (indicating that it was rotated via copy/truncate) - Is this not my scenario? My script is truncating the data and then inserting new data.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 12, 2017, 8:52pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/10 "2017-07-12T20:52:47Z")

</div>

Sigh. But you just said that it just overwrites the file.

With debug logs enabled the file input says a few things about what it's doing. That might contain clues.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [July 20, 2017, 5:53pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/11 "2017-07-20T17:53:38Z")

</div>

Hi Magnus, sorry for the delay.

Here are my logs with debug logs enabled,

13:50:07.889 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:12.890 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:17.356 [[main]\<file] DEBUG logstash.inputs.file - \_globbed\_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]  
13:50:17.890 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:22.890 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:27.889 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:32.359 [[main]\<file] DEBUG logstash.inputs.file - \_globbed\_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]  
13:50:32.888 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:37.887 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:42.886 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:47.363 [[main]\<file] DEBUG logstash.inputs.file - \_globbed\_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]  
13:50:47.885 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:52.885 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:50:57.885 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:02.385 [[main]\<file] DEBUG logstash.inputs.file - \_globbed\_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]  
13:51:02.884 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:07.884 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:12.883 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:17.389 [[main]\<file] DEBUG logstash.inputs.file - \_globbed\_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]  
13:51:17.883 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:22.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:27.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:32.394 [[main]\<file] DEBUG logstash.inputs.file - \_globbed\_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]  
13:51:32.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:37.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:42.881 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:47.398 [[main]\<file] DEBUG logstash.inputs.file - \_globbed\_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]  
13:51:47.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:52.881 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:51:57.881 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:52:02.400 [[main]\<file] DEBUG logstash.inputs.file - \_globbed\_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]  
13:52:02.880 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:52:07.880 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:52:12.879 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline  
13:52:17.404 [[main]\<file] DEBUG logstash.inputs.file - \_globbed\_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [July 25, 2017, 3:13pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/12 "2017-07-25T15:13:18Z")

</div>

Hi Magnus, any update on this please... Please let me know if you need any other details.

---

<div class="post-metadata">

**Author:** ![Patrick\_Stevens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick_stevens/32/20565_2.png) [@Patrick\_Stevens](https://discuss.elastic.co/u/Patrick_Stevens)\
**Post date:** [July 31, 2017, 8:55pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/13 "2017-07-31T20:55:02Z")

</div>

Hi Amruth, were you able to find a solution? I seem to be running into the same problem, and I can't find a clear solution anywhere.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [August 1, 2017, 5:14pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/14 "2017-08-01T17:14:11Z")

</div>

Hi Patrick, I don't have an answer yet!

---

<div class="post-metadata">

**Author:** ![Patrick\_Stevens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick_stevens/32/20565_2.png) [@Patrick\_Stevens](https://discuss.elastic.co/u/Patrick_Stevens)\
**Post date:** [August 1, 2017, 6:01pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/15 "2017-08-01T18:01:42Z")

</div>

Ok, I'll keep you posted if I find anything. I'll give this a few more days, and then I may switch to a different log parser like Fluentd....

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [August 20, 2017, 4:26pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/16 "2017-08-20T16:26:37Z")

</div>

Hi, can someone please help me on this issue?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 17, 2017, 4:27pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-the-log-file-thats-being-updated-automatically-continuously/92686/17 "2017-09-17T16:27:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.