Guys,
its been more than a week i've been struggling with this issue,
if possible, please give it a look and try to help
i have a config file that im running logstash with which is suppose to
fetch the log file i specified in it and stream it to elasticsearch.
problem is that it worked twice and thats it. NO changes made to the file
and most of the times it doest load the data and doesnt show any error msg.
when i change the input from "file" to "stdin' it works fine.
this is the config file, which i belive the syntax is correct since it did
work twice...
Guys,
its been more than a week i've been struggling with this issue,
if possible, please give it a look and try to help
i have a config file that im running logstash with which is suppose to
fetch the log file i specified in it and stream it to elasticsearch.
problem is that it worked twice and thats it. NO changes made to the file
and most of the times it doest load the data and doesnt show any error msg.
when i change the input from "file" to "stdin' it works fine.
this is the config file, which i belive the syntax is correct since it did
work twice...
My recommendation is to use the stdin input in logstash and avoid its file
input. Then, for testing you pipe the file into your logstash instance. But
in production, you should run the GNU version of tail -F (uppercase F
option) to correctly follow all forms of rotated logs, and the pipe that
output into your logstash instance.
I don't know just how robust logstash's file input is, but the GNU version
of tail with the -F option is perfect, so there's no guesswork and no
dependency on hope. Note that even Splunk has a currently open bug with
losing data while trying to follow a rotated file.
Also, I added the multiline processing to the filters; it didn't seem to
work when applied as a stdin codec. Now it works very well together.
The path shows an windows file name, so I am not sure if using tail would
work. On cygwin, there is no -F option, at least on the version I use. On
Linux, the file input works great, especially with rotated file.
I am not a Logstash expert, but I use the file input with the sincedb
option (sincedb_path) and it has worked since day one.
My recommendation is to use the stdin input in logstash and avoid its file
input. Then, for testing you pipe the file into your logstash instance. But
in production, you should run the GNU version of tail -F (uppercase F
option) to correctly follow all forms of rotated logs, and the pipe that
output into your logstash instance.
I don't know just how robust logstash's file input is, but the GNU version
of tail with the -F option is perfect, so there's no guesswork and no
dependency on hope. Note that even Splunk has a currently open bug with
losing data while trying to follow a rotated file.
Also, I added the multiline processing to the filters; it didn't seem to
work when applied as a stdin codec. Now it works very well together.
since im not sure on which side the problem is i posted on both elastic and
logstash groups.
here is the thing:
a week ago, everyhting worked great. i loaded log files from Lstash to ES
and filtered them on the way.
i havent touched none of then for tha past week or so, and when i tried
doing the same operations i had done, using the exact same config files, it
suddenly doesnt work (except for two times that it did and i have no idea
what was different on those times...
as i mentioned, when i change the input to be stdin (and not "file")
everything works fine.
i've added a screen shot of my desktop running elastic and logstash, hope
you'll be able to see what ever it is im not seeing
On Saturday, June 21, 2014 5:42:37 PM UTC+3, Ivan Brusic wrote:
The path shows an windows file name, so I am not sure if using tail would
work. On cygwin, there is no -F option, at least on the version I use. On
Linux, the file input works great, especially with rotated file.
I am not a Logstash expert, but I use the file input with the sincedb
option (sincedb_path) and it has worked since day one.
--
Ivan
On Fri, Jun 20, 2014 at 5:07 PM, Brian <brian....@gmail.com <javascript:>>
wrote:
Eitan,
My recommendation is to use the stdin input in logstash and avoid its
file input. Then, for testing you pipe the file into your logstash
instance. But in production, you should run the GNU version of tail -F
(uppercase F option) to correctly follow all forms of rotated logs, and the
pipe that output into your logstash instance.
I don't know just how robust logstash's file input is, but the GNU
version of tail with the -F option is perfect, so there's no guesswork and
no dependency on hope. Note that even Splunk has a currently open bug with
losing data while trying to follow a rotated file.
Also, I added the multiline processing to the filters; it didn't seem to
work when applied as a stdin codec. Now it works very well together.
Ok, i think i got it this time.
Since_Db.
the problem was that i was uploading the same log file over and over again.
logstash remembers where it stoped reading the file the last time it was
open so it would only update elastic with changes. since i deleted the data
from elastic, and the log file i loaded didnt change from the last time it
was loaded nothing happend. Thanks for trying to help!
On Saturday, June 21, 2014 1:02:33 AM UTC+3, Eitan Vesely wrote:
Guys,
its been more than a week i've been struggling with this issue,
if possible, please give it a look and try to help
i have a config file that im running logstash with which is suppose to
fetch the log file i specified in it and stream it to elasticsearch.
problem is that it worked twice and thats it. NO changes made to the file
and most of the times it doest load the data and doesnt show any error msg.
when i change the input from "file" to "stdin' it works fine.
this is the config file, which i belive the syntax is correct since it did
work twice...
One more question: We have two different forms of rotated files (on *IX
systems; no Windows servers):
Standard log4j rotation: The XXX.log file is renamed to XXX-.log
and a new XXX.log file is created. The name doesn't change, but the inode
changes.
When we switched many of our applications to use log4j2, we don't rotate
the log files using log4j2. Instead, we have a cron job that, once per
hour, makes a copy of the XXX.log file and then truncates the XXX.log file;
in the background it compresses the copy. In this case, the name doesn't
change, the inode doesn't change, but the size suddenly drops to 0 before
it starts filling again from the beginning.
The GNU tail -F command handles both of these equally perfectly. Does
logstash also handle both of these cases? Thanks in advance!
P.S. I am not a logstash expert either, but it's been a lot of fun to
rediscover Elasticsearch from the ELK perspective (auto-mapping,
auto-creation of indices, and so on).
Brian
On Saturday, June 21, 2014 10:42:37 AM UTC-4, Ivan Brusic wrote:
The path shows an windows file name, so I am not sure if using tail would
work. On cygwin, there is no -F option, at least on the version I use. On
Linux, the file input works great, especially with rotated file.
I am not a Logstash expert, but I use the file input with the sincedb
option (sincedb_path) and it has worked since day one.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
