# Logstash doesn't read changes in file and need to start from bin every time

**URL:** <https://discuss.elastic.co/t/logstash-doesnt-read-changes-in-file-and-need-to-start-from-bin-every-time/172247>\
**Category:** Logstash\
**Created:** [March 14, 2019, 5:23am UTC](https://discuss.elastic.co/t/logstash-doesnt-read-changes-in-file-and-need-to-start-from-bin-every-time/172247 "2019-03-14T05:23:18Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sukarn001](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukarn001/32/49865_2.png) [@sukarn001](https://discuss.elastic.co/u/sukarn001)\
**Post date:** [March 14, 2019, 5:23am UTC](https://discuss.elastic.co/t/logstash-doesnt-read-changes-in-file-and-need-to-start-from-bin-every-time/172247/1 "2019-03-14T05:23:18Z")

</div>

Hello,

I have installed Logstash 6.6.1 in rhel 7, from RPM.  
Now I have created a file under /etc/logstash/conf.d/ **apache.conf**. This file keeps updating itself.  
I ran following command to send logs to elasticsearch:  
./logstash -f /etc/logstash/conf.d/apache.conf

It works fine, i can see the logs till command run time. However next time the file is updated, i don't see any changes in elasticsearch.  
I have to run above command every time to see updated logs in elasticsearch.  
Is there any way to automatic reload logstash, so that i can see updated logs?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 14, 2019, 5:55am UTC](https://discuss.elastic.co/t/logstash-doesnt-read-changes-in-file-and-need-to-start-from-bin-every-time/172247/2 "2019-03-14T05:55:46Z")

</div>

Logstash tails files so as long as new data is appended to the file it should be read. If existing content however is modified this is unlikely to get picked up. What type of data does the file contain? How is data updated?

---

<div class="post-metadata">

**Author:** ![sukarn001](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukarn001/32/49865_2.png) [@sukarn001](https://discuss.elastic.co/u/sukarn001)\
**Post date:** [March 14, 2019, 6:14am UTC](https://discuss.elastic.co/t/logstash-doesnt-read-changes-in-file-and-need-to-start-from-bin-every-time/172247/3 "2019-03-14T06:14:25Z")

</div>

this file contains HTTPD access logs, i am using rsync to sync logs from a cloud server, as my elasticsearch server is not public.here is the content of file:

`input  
{  
file {  
path =\> "/root/Desktop/access\_log"  
type =\> "logs"  
start\_position =\> "beginning"  
}  
}  
filter  
{  
grok {  
match =\> {  
"message" =\> "%{COMBINEDAPACHELOG}"  
}  
}  
mutate {  
convert =\> {"bytes" =\> "integer" }  
}  
date{  
match =\> ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]  
locale =\> en  
remove\_field =\> "timestamp"  
}  
geoip {  
source =\> "clientip"  
}  
useragent {  
source =\> "agent"  
target =\> "useragent"  
}  
}  
output{  
elasticsearch {  
hosts =\> "[http://localhost:9200](http://localhost:9200)"  
index =\> "mysite"  
document\_type =\> "usagereport"  
}

stdout {}

}`

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 14, 2019, 6:47am UTC](https://discuss.elastic.co/t/logstash-doesnt-read-changes-in-file-and-need-to-start-from-bin-every-time/172247/4 "2019-03-14T06:47:59Z")

</div>

Using raunchy to update files is not recommended and I would expect you to end up with lots of duplicates as each update would give a new inode which Filebeat would interpret as a new file and read from the beginning.

---

<div class="post-metadata">

**Author:** ![sukarn001](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukarn001/32/49865_2.png) [@sukarn001](https://discuss.elastic.co/u/sukarn001)\
**Post date:** [March 14, 2019, 9:18am UTC](https://discuss.elastic.co/t/logstash-doesnt-read-changes-in-file-and-need-to-start-from-bin-every-time/172247/5 "2019-03-14T09:18:43Z")

</div>

Hi,

When I re-run same logstash command I do not get any duplicates, its just how logstash will read updated file? or whenever changes are being done on file. I am not using Filebeat right now.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 14, 2019, 9:33am UTC](https://discuss.elastic.co/t/logstash-doesnt-read-changes-in-file-and-need-to-start-from-bin-every-time/172247/6 "2019-03-14T09:33:42Z")

</div>

Logstash file input as far as I know behaves the same way, so I would expect similar behaviour. Are you rsyncing the full and expanding file each time?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 11, 2019, 9:33am UTC](https://discuss.elastic.co/t/logstash-doesnt-read-changes-in-file-and-need-to-start-from-bin-every-time/172247/7 "2019-04-11T09:33:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.