Logstash max_open_files setting not working?

[jspriesterbach (James Spriesterbach)]

Hello,

I am using logstash version 6.3 and I can't seem to get the max_open_files to stick. According to the proc it is set properly:

root@ip-172-31-47-78:/proc/1433# cat limits | grep "Max open files"
Max open files 16384 16384 files

But in the logs I am still getting:

[2018-08-06T19:12:48,088][WARN ][logstash.inputs.file ] Reached open files limit: 4095, set by the 'max_open_files' option or default, files yet to open: 14325

in /etc/logstash/startup.options LS_OPEN_FILES is set to 16384:

startup.options:LS_OPEN_FILES=16384

I also tried setting it in /etc/default/logstash

/etc/logstash# cat /etc/default/logstash |grep OPEN
LS_OPEN_FILES="16384"
MAX_OPEN_FILES="16384"

Any help is appreciated.

[Badger]

You need to set it on the file input itself

input {
    file {
        path => ...
        max_open_files => 16384
        ...

[jspriesterbach (James Spriesterbach)]

That does not appear to work either. I have set this and its still throwing the 4096 limit error:

/home/ubuntu# cat /etc/logstash/conf.d/rds_slow_logs.conf
input {
file {
max_open_files => 20000
path => "/mnt/baks/rds_slowquery_logs/*"
type => "rds_slow_query"
start_position => "beginning"
}
}

[Badger]

Does setting the environment variable FILEWATCH_MAX_OPEN_FILES work?

[jspriesterbach (James Spriesterbach)]

I set it in /etc/logstash/startup.options but does not appear to do anything either. I am using a debian based system (aws ec2, stretch/sid)

[tetra1 (boni)]

hello. can logstash insert txt file into elasticsearch?