# Issues with "file" input from logstash to elastic - please read

**URL:** <https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239>\
**Category:** Elasticsearch\
**Created:** [June 20, 2014, 10:02pm UTC](https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239 "2014-06-20T22:02:33Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eitan\_Vesely](https://avatars.discourse-cdn.com/v4/letter/e/82dd89/32.png) [@Eitan\_Vesely](https://discuss.elastic.co/u/Eitan_Vesely)\
**Post date:** [June 20, 2014, 10:02pm UTC](https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239/1 "2014-06-20T22:02:33Z")

</div>

Guys,  
its been more than a week i've been struggling with this issue,  
if possible, please give it a look and try to help ☹

i have a config file that im running logstash with which is suppose to  
fetch the log file i specified in it and stream it to elasticsearch.

problem is that it worked twice and thats it. NO changes made to the file  
and most of the times it doest load the data and doesnt show any error msg.  
when i change the input from "file" to "stdin' it works fine.

this is the config file, which i belive the syntax is correct since it did  
work twice...

input{  
file{  
path =\> "C:\elasticsearch-1.2.0\testLog.txt"  
start\_position =\> "beginning"  
}  
}  
output{  
elasticsearch{  
host=\> "localhost"  
index=\> "tester3"  
protocol=\> "http"  
}  
}

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8b0634eb-dd2c-47f3-9959-2e48bdcc349d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8b0634eb-dd2c-47f3-9959-2e48bdcc349d%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 20, 2014, 10:24pm UTC](https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239/2 "2014-06-20T22:24:40Z")

</div>

You'll have better luck sending this to the Logstash mailing list 🙂

Regards,  
Mark Walkom

Infrastructure Engineer  
Campaign Monitor  
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)

On 21 June 2014 08:02, Eitan Vesely [eitanjnk@gmail.com](mailto:eitanjnk@gmail.com) wrote:

> Guys,  
> its been more than a week i've been struggling with this issue,  
> if possible, please give it a look and try to help ☹
> 
> i have a config file that im running logstash with which is suppose to  
> fetch the log file i specified in it and stream it to elasticsearch.
> 
> problem is that it worked twice and thats it. NO changes made to the file  
> and most of the times it doest load the data and doesnt show any error msg.  
> when i change the input from "file" to "stdin' it works fine.
> 
> this is the config file, which i belive the syntax is correct since it did  
> work twice...
> 
> input{  
> file{  
> path =\> "C:\elasticsearch-1.2.0\testLog.txt"  
> start\_position =\> "beginning"  
> }  
> }  
> output{  
> elasticsearch{  
> host=\> "localhost"  
> index=\> "tester3"  
> protocol=\> "http"  
> }  
> }
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/8b0634eb-dd2c-47f3-9959-2e48bdcc349d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8b0634eb-dd2c-47f3-9959-2e48bdcc349d%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/8b0634eb-dd2c-47f3-9959-2e48bdcc349d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8b0634eb-dd2c-47f3-9959-2e48bdcc349d%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YhwCh2XQ1BjK5c5czTy3t0Wa%3DK46st6Gr5Ei%3D5JAkCyg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YhwCh2XQ1BjK5c5czTy3t0Wa%3DK46st6Gr5Ei%3D5JAkCyg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\
**Post date:** [June 21, 2014, 12:07am UTC](https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239/3 "2014-06-21T00:07:57Z")

</div>

Eitan,

My recommendation is to use the stdin input in logstash and avoid its file  
input. Then, for testing you pipe the file into your logstash instance. But  
in production, you should run the GNU version of _tail -F_ (uppercase F  
option) to correctly follow all forms of rotated logs, and the pipe that  
output into your logstash instance.

I don't know just how robust logstash's file input is, but the GNU version  
of tail with the -F option is perfect, so there's no guesswork and no  
dependency on hope. Note that even Splunk has a currently open bug with  
losing data while trying to follow a rotated file.

Also, I added the multiline processing to the filters; it didn't seem to  
work when applied as a stdin codec. Now it works very well together.

Anyway, that's what our group is doing.

And yes, the logstash-users  
[https://groups.google.com/forum/#!forum/logstash-users](https://groups.google.com/forum/#!forum/logstash-users) group is also  
rather active and is a good place for logstash-specific help.

Brian

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [June 21, 2014, 2:42pm UTC](https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239/4 "2014-06-21T14:42:21Z")

</div>

The path shows an windows file name, so I am not sure if using tail would  
work. On cygwin, there is no -F option, at least on the version I use. On  
Linux, the file input works great, especially with rotated file.

I am not a Logstash expert, but I use the file input with the sincedb  
option (sincedb\_path) and it has worked since day one.

--  
Ivan

On Fri, Jun 20, 2014 at 5:07 PM, Brian [brian.from.fl@gmail.com](mailto:brian.from.fl@gmail.com) wrote:

> Eitan,
> 
> My recommendation is to use the stdin input in logstash and avoid its file  
> input. Then, for testing you pipe the file into your logstash instance. But  
> in production, you should run the GNU version of _tail -F_ (uppercase F  
> option) to correctly follow all forms of rotated logs, and the pipe that  
> output into your logstash instance.
> 
> I don't know just how robust logstash's file input is, but the GNU version  
> of tail with the -F option is perfect, so there's no guesswork and no  
> dependency on hope. Note that even Splunk has a currently open bug with  
> losing data while trying to follow a rotated file.
> 
> Also, I added the multiline processing to the filters; it didn't seem to  
> work when applied as a stdin codec. Now it works very well together.
> 
> Anyway, that's what our group is doing.
> 
> And yes, the logstash-users  
> [https://groups.google.com/forum/#!forum/logstash-users](https://groups.google.com/forum/#!forum/logstash-users) group is also  
> rather active and is a good place for logstash-specific help.
> 
> Brian
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBxiUi%2BUNuyHUN9vM5z0bPsFyCYV%2BeZV3C1r5jzApOJww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBxiUi%2BUNuyHUN9vM5z0bPsFyCYV%2BeZV3C1r5jzApOJww%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Eitan\_Vesely](https://avatars.discourse-cdn.com/v4/letter/e/82dd89/32.png) [@Eitan\_Vesely](https://discuss.elastic.co/u/Eitan_Vesely)\
**Post date:** [June 21, 2014, 5:11pm UTC](https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239/5 "2014-06-21T17:11:48Z")

</div>

Thanks Guys.

since im not sure on which side the problem is i posted on both elastic and  
logstash groups.

here is the thing:  
a week ago, everyhting worked great. i loaded log files from Lstash to ES  
and filtered them on the way.  
i havent touched none of then for tha past week or so, and when i tried  
doing the same operations i had done, using the exact same config files, it  
suddenly doesnt work (except for two times that it did and i have no idea  
what was different on those times...

as i mentioned, when i change the input to be stdin (and not "file")  
everything works fine.

i've added a screen shot of my desktop running elastic and logstash, hope  
you'll be able to see what ever it is im not seeing

On Saturday, June 21, 2014 5:42:37 PM UTC+3, Ivan Brusic wrote:

> The path shows an windows file name, so I am not sure if using tail would  
> work. On cygwin, there is no -F option, at least on the version I use. On  
> Linux, the file input works great, especially with rotated file.
> 
> I am not a Logstash expert, but I use the file input with the sincedb  
> option (sincedb\_path) and it has worked since day one.
> 
> --  
> Ivan
> 
> On Fri, Jun 20, 2014 at 5:07 PM, Brian \<[brian....@gmail.com](mailto:brian....@gmail.com) \<javascript:\>\>  
> wrote:
> 
> > Eitan,
> > 
> > My recommendation is to use the stdin input in logstash and avoid its  
> > file input. Then, for testing you pipe the file into your logstash  
> > instance. But in production, you should run the GNU version of _tail -F_  
> > (uppercase F option) to correctly follow all forms of rotated logs, and the  
> > pipe that output into your logstash instance.
> > 
> > I don't know just how robust logstash's file input is, but the GNU  
> > version of tail with the -F option is perfect, so there's no guesswork and  
> > no dependency on hope. Note that even Splunk has a currently open bug with  
> > losing data while trying to follow a rotated file.
> > 
> > Also, I added the multiline processing to the filters; it didn't seem to  
> > work when applied as a stdin codec. Now it works very well together.
> > 
> > Anyway, that's what our group is doing.
> > 
> > And yes, the logstash-users  
> > [https://groups.google.com/forum/#!forum/logstash-users](https://groups.google.com/forum/#!forum/logstash-users) group is also  
> > rather active and is a good place for logstash-specific help.
> > 
> > Brian
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .
> > 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dd20b87-c19c-4c77-b1bf-ea30b1156372%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dd20b87-c19c-4c77-b1bf-ea30b1156372%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Eitan\_Vesely](https://avatars.discourse-cdn.com/v4/letter/e/82dd89/32.png) [@Eitan\_Vesely](https://discuss.elastic.co/u/Eitan_Vesely)\
**Post date:** [June 21, 2014, 8:22pm UTC](https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239/6 "2014-06-21T20:22:37Z")

</div>

Ok, i think i got it this time.  
Since\_Db.  
the problem was that i was uploading the same log file over and over again.  
logstash remembers where it stoped reading the file the last time it was  
open so it would only update elastic with changes. since i deleted the data  
from elastic, and the log file i loaded didnt change from the last time it  
was loaded nothing happend. Thanks for trying to help!

On Saturday, June 21, 2014 1:02:33 AM UTC+3, Eitan Vesely wrote:

> Guys,  
> its been more than a week i've been struggling with this issue,  
> if possible, please give it a look and try to help ☹
> 
> i have a config file that im running logstash with which is suppose to  
> fetch the log file i specified in it and stream it to elasticsearch.
> 
> problem is that it worked twice and thats it. NO changes made to the file  
> and most of the times it doest load the data and doesnt show any error msg.  
> when i change the input from "file" to "stdin' it works fine.
> 
> this is the config file, which i belive the syntax is correct since it did  
> work twice...
> 
> input{  
> file{  
> path =\> "C:\elasticsearch-1.2.0\testLog.txt"  
> start\_position =\> "beginning"  
> }  
> }  
> output{  
> elasticsearch{  
> host=\> "localhost"  
> index=\> "tester3"  
> protocol=\> "http"  
> }  
> }

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a15b3d77-2064-433f-8dc7-c00f6f44a89e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a15b3d77-2064-433f-8dc7-c00f6f44a89e%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\
**Post date:** [June 23, 2014, 2:01am UTC](https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239/7 "2014-06-23T02:01:36Z")

</div>

Thanks so much for the feedback, Ivan.

One more question: We have two different forms of rotated files (on \*IX  
systems; no Windows servers):

1. Standard log4j rotation: The XXX.log file is renamed to XXX-.log  
and a new XXX.log file is created. The name doesn't change, but the inode  
changes.
2. When we switched many of our applications to use log4j2, we don't rotate  
the log files using log4j2. Instead, we have a cron job that, once per  
hour, makes a copy of the XXX.log file and then truncates the XXX.log file;  
in the background it compresses the copy. In this case, the name doesn't  
change, the inode doesn't change, but the size suddenly drops to 0 before  
it starts filling again from the beginning.

The GNU tail -F command handles both of these equally perfectly. Does  
logstash also handle both of these cases? Thanks in advance!

P.S. I am not a logstash expert either, but it's been a lot of fun to  
rediscover Elasticsearch from the ELK perspective (auto-mapping,  
auto-creation of indices, and so on).

Brian

On Saturday, June 21, 2014 10:42:37 AM UTC-4, Ivan Brusic wrote:

> The path shows an windows file name, so I am not sure if using tail would  
> work. On cygwin, there is no -F option, at least on the version I use. On  
> Linux, the file input works great, especially with rotated file.
> 
> I am not a Logstash expert, but I use the file input with the sincedb  
> option (sincedb\_path) and it has worked since day one.
> 
> --  
> Ivan

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9f1433e1-748e-4a20-980f-5112a1f965fa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f1433e1-748e-4a20-980f-5112a1f965fa%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:20am UTC](https://discuss.elastic.co/t/issues-with-file-input-from-logstash-to-elastic-please-read/18239/8 "2017-07-06T01:20:40Z")

</div>