File input not working - logstash 2.2.2

Elastic Stack Logstash
#1

Hi

First of all, sorry for my poor english, I will try to explain my problem as better as I can.

I have just begun to use logstash + elasticsearch and I have a problem with file input.
Each program is running in a virtual machine (CentOS7) and everything work except when i must load logs from a file!
Data are perfectly processed by logstash and stored by elasticsearch when I insert them manually, or also when logstash is listening for TCP and UDP event.

This is the simple program that I'm using as test

input {
file {
path => "/tmp/tutorial.log"
start_position => beginning
}
}

filter {
grok { match => { "message" => "%{COMBINEDAPACHELOG}"} }
geoip { source => "clientip" }
}

output {
elasticsearch { hosts => "192.168.122.243:9200"}
stdout { codec => rubydebug }
}

The result is the follow:

Settings: Default pipeline workers: 1
Logstash startup completed

Nothing else, I have alredy try to use the --debug option but nothing, all looks fine :confused:

The file exist and is in the correct folder. As a test, I have write a similar code but using stdinput {} and I lunch the program using the follow command and it works, all the data are normalized and stored in the database!

cat /tmp/tutorial.log | bin/logstash -f test.conf

Thanks in advance for your help.

Giulio

#2

Logstash is waiting for additional input to be written to /tmp/tutorial.log since it at some point has processed the file before (or you just added start_position => beginning). Please read the file input documentation's section on sincedb files.

If you want to reprocess /tmp/tutorial.log multiple times I suggest you also set sincedb_path => "/dev/null" so that Logstash doesn't save and remember its current position in the file.

This is the third time I see this question today so you'll find several other answers in the archives.

#3

Thanks for your reply.

Maybe I haven't explain my problem properly or simply logstash is driving me crazy (problably both options)
I must not write the data on a file, I must read from file and write normalized them on DB (elasticsearch).
As I write, all work perfectly using other forms of input (syslog, direct input, ecc) but when I load the data from file simply nothing happen.

I've alredy read the input option guide (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html) but it doesn't help much.

I'm sorry to disturb you and the community but after a couple of days fighting against this (stupid) problem but I really don't have idea where else I can ask for help.

Thank you again

#4

Did you attempt to delete the sincedb file or set sincedb_path to /dev/null?

Another cause for this could be that the file is older than 24 hours. By default such old files will be ignored. See the file input's ignore_older option.

#5

I've already try both options and nothing... Anything works...
At this point I simply surrender.
I thank you again for your kind answer.

Have a nice day

Giulio

#6