Issues with "file" input from logstash to elastic - please read


(Eitan Vesely) #1

Guys,
its been more than a week i've been struggling with this issue,
if possible, please give it a look and try to help :frowning:

i have a config file that im running logstash with which is suppose to
fetch the log file i specified in it and stream it to elasticsearch.

problem is that it worked twice and thats it. NO changes made to the file
and most of the times it doest load the data and doesnt show any error msg.
when i change the input from "file" to "stdin' it works fine.

this is the config file, which i belive the syntax is correct since it did
work twice...

input{
file{
path => "C:\elasticsearch-1.2.0\testLog.txt"
start_position => "beginning"
}
}
output{
elasticsearch{
host=> "localhost"
index=> "tester3"
protocol=> "http"
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8b0634eb-dd2c-47f3-9959-2e48bdcc349d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

You'll have better luck sending this to the Logstash mailing list :slight_smile:

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 21 June 2014 08:02, Eitan Vesely eitanjnk@gmail.com wrote:

Guys,
its been more than a week i've been struggling with this issue,
if possible, please give it a look and try to help :frowning:

i have a config file that im running logstash with which is suppose to
fetch the log file i specified in it and stream it to elasticsearch.

problem is that it worked twice and thats it. NO changes made to the file
and most of the times it doest load the data and doesnt show any error msg.
when i change the input from "file" to "stdin' it works fine.

this is the config file, which i belive the syntax is correct since it did
work twice...

input{
file{
path => "C:\elasticsearch-1.2.0\testLog.txt"
start_position => "beginning"
}
}
output{
elasticsearch{
host=> "localhost"
index=> "tester3"
protocol=> "http"
}
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8b0634eb-dd2c-47f3-9959-2e48bdcc349d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8b0634eb-dd2c-47f3-9959-2e48bdcc349d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YhwCh2XQ1BjK5c5czTy3t0Wa%3DK46st6Gr5Ei%3D5JAkCyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Brian Yoder) #3

Eitan,

My recommendation is to use the stdin input in logstash and avoid its file
input. Then, for testing you pipe the file into your logstash instance. But
in production, you should run the GNU version of tail -F (uppercase F
option) to correctly follow all forms of rotated logs, and the pipe that
output into your logstash instance.

I don't know just how robust logstash's file input is, but the GNU version
of tail with the -F option is perfect, so there's no guesswork and no
dependency on hope. Note that even Splunk has a currently open bug with
losing data while trying to follow a rotated file.

Also, I added the multiline processing to the filters; it didn't seem to
work when applied as a stdin codec. Now it works very well together.

Anyway, that's what our group is doing.

And yes, the logstash-users
https://groups.google.com/forum/#!forum/logstash-users group is also
rather active and is a good place for logstash-specific help.

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #4

The path shows an windows file name, so I am not sure if using tail would
work. On cygwin, there is no -F option, at least on the version I use. On
Linux, the file input works great, especially with rotated file.

I am not a Logstash expert, but I use the file input with the sincedb
option (sincedb_path) and it has worked since day one.

--
Ivan

On Fri, Jun 20, 2014 at 5:07 PM, Brian brian.from.fl@gmail.com wrote:

Eitan,

My recommendation is to use the stdin input in logstash and avoid its file
input. Then, for testing you pipe the file into your logstash instance. But
in production, you should run the GNU version of tail -F (uppercase F
option) to correctly follow all forms of rotated logs, and the pipe that
output into your logstash instance.

I don't know just how robust logstash's file input is, but the GNU version
of tail with the -F option is perfect, so there's no guesswork and no
dependency on hope. Note that even Splunk has a currently open bug with
losing data while trying to follow a rotated file.

Also, I added the multiline processing to the filters; it didn't seem to
work when applied as a stdin codec. Now it works very well together.

Anyway, that's what our group is doing.

And yes, the logstash-users
https://groups.google.com/forum/#!forum/logstash-users group is also
rather active and is a good place for logstash-specific help.

Brian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBxiUi%2BUNuyHUN9vM5z0bPsFyCYV%2BeZV3C1r5jzApOJww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Eitan Vesely) #5

Thanks Guys.

since im not sure on which side the problem is i posted on both elastic and
logstash groups.

here is the thing:
a week ago, everyhting worked great. i loaded log files from Lstash to ES
and filtered them on the way.
i havent touched none of then for tha past week or so, and when i tried
doing the same operations i had done, using the exact same config files, it
suddenly doesnt work (except for two times that it did and i have no idea
what was different on those times...

as i mentioned, when i change the input to be stdin (and not "file")
everything works fine.

i've added a screen shot of my desktop running elastic and logstash, hope
you'll be able to see what ever it is im not seeing

On Saturday, June 21, 2014 5:42:37 PM UTC+3, Ivan Brusic wrote:

The path shows an windows file name, so I am not sure if using tail would
work. On cygwin, there is no -F option, at least on the version I use. On
Linux, the file input works great, especially with rotated file.

I am not a Logstash expert, but I use the file input with the sincedb
option (sincedb_path) and it has worked since day one.

--
Ivan

On Fri, Jun 20, 2014 at 5:07 PM, Brian <brian....@gmail.com <javascript:>>
wrote:

Eitan,

My recommendation is to use the stdin input in logstash and avoid its
file input. Then, for testing you pipe the file into your logstash
instance. But in production, you should run the GNU version of tail -F
(uppercase F option) to correctly follow all forms of rotated logs, and the
pipe that output into your logstash instance.

I don't know just how robust logstash's file input is, but the GNU
version of tail with the -F option is perfect, so there's no guesswork and
no dependency on hope. Note that even Splunk has a currently open bug with
losing data while trying to follow a rotated file.

Also, I added the multiline processing to the filters; it didn't seem to
work when applied as a stdin codec. Now it works very well together.

Anyway, that's what our group is doing.

And yes, the logstash-users
https://groups.google.com/forum/#!forum/logstash-users group is also
rather active and is a good place for logstash-specific help.

Brian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9bbe59f4-93f1-4b59-8258-89301a8c5469%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6dd20b87-c19c-4c77-b1bf-ea30b1156372%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Eitan Vesely) #6

Ok, i think i got it this time.
Since_Db.
the problem was that i was uploading the same log file over and over again.
logstash remembers where it stoped reading the file the last time it was
open so it would only update elastic with changes. since i deleted the data
from elastic, and the log file i loaded didnt change from the last time it
was loaded nothing happend. Thanks for trying to help!

On Saturday, June 21, 2014 1:02:33 AM UTC+3, Eitan Vesely wrote:

Guys,
its been more than a week i've been struggling with this issue,
if possible, please give it a look and try to help :frowning:

i have a config file that im running logstash with which is suppose to
fetch the log file i specified in it and stream it to elasticsearch.

problem is that it worked twice and thats it. NO changes made to the file
and most of the times it doest load the data and doesnt show any error msg.
when i change the input from "file" to "stdin' it works fine.

this is the config file, which i belive the syntax is correct since it did
work twice...

input{
file{
path => "C:\elasticsearch-1.2.0\testLog.txt"
start_position => "beginning"
}
}
output{
elasticsearch{
host=> "localhost"
index=> "tester3"
protocol=> "http"
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a15b3d77-2064-433f-8dc7-c00f6f44a89e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Brian Yoder) #7

Thanks so much for the feedback, Ivan.

One more question: We have two different forms of rotated files (on *IX
systems; no Windows servers):

  1. Standard log4j rotation: The XXX.log file is renamed to XXX-.log
    and a new XXX.log file is created. The name doesn't change, but the inode
    changes.
  2. When we switched many of our applications to use log4j2, we don't rotate
    the log files using log4j2. Instead, we have a cron job that, once per
    hour, makes a copy of the XXX.log file and then truncates the XXX.log file;
    in the background it compresses the copy. In this case, the name doesn't
    change, the inode doesn't change, but the size suddenly drops to 0 before
    it starts filling again from the beginning.

The GNU tail -F command handles both of these equally perfectly. Does
logstash also handle both of these cases? Thanks in advance!

P.S. I am not a logstash expert either, but it's been a lot of fun to
rediscover Elasticsearch from the ELK perspective (auto-mapping,
auto-creation of indices, and so on).

Brian

On Saturday, June 21, 2014 10:42:37 AM UTC-4, Ivan Brusic wrote:

The path shows an windows file name, so I am not sure if using tail would
work. On cygwin, there is no -F option, at least on the version I use. On
Linux, the file input works great, especially with rotated file.

I am not a Logstash expert, but I use the file input with the sincedb
option (sincedb_path) and it has worked since day one.

--
Ivan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f1433e1-748e-4a20-980f-5112a1f965fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #8