Problem input file version 2.2.0

Elastic Stack Logstash
1

Maurizio Maltese: Hello there everybody
I recently updated to the last version of Logstash, taking it to the 2.2.0.
I'm having some issues with the input file plugin. When I run Logstash and insert files that have to be uploaded to ElasticSearch in the relative folder, Logstash doesn't process all the files but one, or sometimes not even that one.
The only way to make Logstash process files is to open and save them. Could you explain why this behaviour is different from the one in version 1.5.5?
thank you in advance

2

I experience the same problem after I updated to Logstash 2.2.1. This was not an issue when using Logstash 2.1.0. I would like to know how to solve this.

Thank you in advance.

3

@chicco_95 @DaanCreemers

Folks,

Please try putting the value beginning in double quote for start position in input config.

Eg.

input {
  file {
    path => "/home/data/your-log-file.txt"
    start_position => "beginning"
  }
}

I spent 6 hours pulling my hair for this. It works after I added start_position the value in the double quotes. Hope it works for you.

Cheers.

4

@yett @DaanCreemers @chicco_95

Hi,

I face a the same issue with Logstash version 2.2.2-1 input (file plugin), this issue appared in version 2.x.x for us it seems.
We've tried Yett's fix but that does not solve it.

After uploading a file to log directory it won't proccess the file, the only way is to run touch filename.txt so the date changes and thus Logstash will start prosessing the file with no problem.

This happens both in production and in a clean test enviroment.
To clarify its been tested in production and on a clean install of CentOS 7 to see if the problem could be reproduced.

Anyone out there have a solution for this?

System info.
CentOS Linux 7, Oracle JDK 8u66 and Logstash 2.2.2 (Repo install RPM 2.2.2-1), ES 2.2.X.

Cheers,
Kevin

5

Hi again,

I've found out what went "wrong", the default behavior of Logstash file input plugin seems to be ignoring files that are older than 86.400 seconds (24h) which explains why changing dates of the old files with touch filename.txt worked.
This seems to be the default behavior for Logstash 2.2.X? as I've only observed the problem with Logstash 2.2.x.

Found this out by running debugging on Logstash with logstash -f /path/to/config.conf --debug, which pointed me in the right direction.
See documentation for details

I've now added ignore_older => 0 to the file {} configuration and importing of older files is working as intended again.

Hope it helps if someone else is struggeling with stalled imports of files over 24h.

Cheers,
Kevin

2 Likes
6

Thank you!

1 Like
7

Thank you @kevinava. The ignore_older option was indeed the problem as this option was added in logstash 2.2.0.

1 Like