Reading a log file into Logstash

Elastic Stack Logstash
1

I'm new to logstash/elasticsearch and am trying to teach myself how to use them. I've written a config file to read a file into logstash and then forward it to elasticsearch and, for the life of me, logstash never seems to read the file. Here is the config:

input {
file {
path => "C:\Users\baneling\Desktop\logstash-1.5.1\bin\test.log"
}
}
output {
stdout { codec => rubydebug }
#elasticsearch{ host => localhost }
}

The contents of test.log are the following:

this is a test 1
this is a test 2
this is a test 3
this is a test 4

I've tried configuring it with a variety of types (apache-access, apache, log4j). I've tried using grok patterns to look for stuff to take from the lines in the file. I've specified directly the sincedb path )the default is correct, though). I've even tried using the start position as "beginning" and have tried modifying the file after logstash is running since it is essentially tailing it. Nothing seems to work aside from the stdin plugin - any advice? I'm on a Windows 8 64bit OS, by the way. Thanks.

1 Like
2

It's a sincedb issue, check the file input docs for more info on it. But the short of it is that you need to delete the file that tracks Logstash's progress in processing it.

3

Hi Warkolm,

I've tried deleting the sincedb files and then starting logstash fresh - that didn't seem work work either. I'll do some more reading, but any other, more specific suggestions are welcome!

4

Then it's probably https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-start_position

5

Hi Warkolm,

I juststopped logstash, deleted my sincedb file, modified my config to the one below, and started logstash. Still not working. Any more suggestions?

input {
file {
path => "C:\Users\grusz_000\Desktop\logstash-1.5.1\bin\ip.log"
start_position => "beginning"
}
}
output {
stdout { codec => rubydebug }
#elasticsearch{ host => localhost }
}

6

A permissions issue? Try starting Logstash with --verbose or even --debug and see what it says. If it doesn't scream about the permissions it'll tell you straight out what it thinks about sincedb and the current position in the file.

7

Hey

Just confirming the obvious, here. Are you not seeing output on stdout?

Did you try enabling the elasticsearch output and checking there as well?

J

8

Hi Everyone,

It wasn't any kind of permission issue or anything. I fixed it by adding in the following:

sincedb_path > "/dev/null"

I don't quite know why this lets me read from a file. Can anyone explain?

9

I can't explain it either, but having a null sincedb_path is a very bad idea since Logstash won't be able to keep track of the current position in each file, so you may end up processing files more than once or miss data. I maintain that starting Logstash with --verbose or --debug will divulge something interesting (probably about the sincedb_files since that's where its hang-up seems to be).

2 Likes