I understand this question with the title has been asked many times before and almost every thread i has opened and learned here, but the doesn't come across any concrete answer so far specially for the Linux with systemd service. I have got the below errors:
[2019-02-06T07:47:01,817][WARN ][filewatch.tailmode.processor] Reached open files limit: 4095, set by the 'max_open_files' option or default, files yet to open: 422
[2019-02-06T07:47:22,314][WARN ][filewatch.tailmode.processor] Reached open files limit: 4095, set by the 'max_open_files' option or default, files yet to open: 422
[2019-02-06T07:47:43,927][WARN ][filewatch.tailmode.processor] Reached open files limit: 4095, set by the 'max_open_files' option or default, files yet to open: 422
[2019-02-06T07:48:04,406][WARN ][filewatch.tailmode.processor] Reached open files limit: 4095, set by the 'max_open_files' option or default, files yet to open: 422
[2019-02-06T07:48:25,934][WARN ][filewatch.tailmode.processor] Reached open files limit: 4095, set by the 'max_open_files' option or default, files yet to open: 422
[2019-02-06T07:48:46,529][WARN ][filewatch.tailmode.processor] Reached open files limit: 4095, set by the 'max_open_files' option or default, files yet to open: 422
I understand this is generic logs and dictates its meaning, However i have followed all the document and threads as possible to fix my issue before coming here to post hence i'm concluding all the Steps i have taken so far to fix the problem but did not worked.
-
limit defined in the limits.conf file for root and wide open to all as `*
`*` soft nofile 64000 `*` hard nofile 64000 root soft nofile 64000 root hard nofile 64000 -
Max file defined in the main config file sysctl.conf to system-wide implementation:
[els_hosts] # cat /etc/sysctl.conf net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 vm.max_map_count = 262144 fs.file-max = 655360
- As the system is running with RHEL7 with systemd Service hence also applied the changes under service.d/ dir path.
[els_hosts] # cat /etc/systemd/system/logstash.service.d/logstashlimit.conf [Service] LimitNOFILE=256000 LimitMEMLOCK=infinity
Verifying the limits under /proc , ulimit and logstash's startup.options as follows:
[els_hosts] # cat /proc/sys/fs/file-max 655360
> [els_hosts] # ulimit -a | grep open
> open files (-n) 64000
[els_hosts] # ulimit -Hn 64000
[els_hosts] # ulimit -Sn 64000
[els_hosts] # grep "LS_OPEN_FILES=16384" /etc/logstash/startup.options LS_OPEN_FILES=16384
am i missing something apart from these? I really taken all them the time but couldn't figure it out yet.
OS details: RHEL 7
Logstash version: 6.5.x
My logstash.conf file:
input {
file {
path => [ "/data/rmlogs_SJ/*.txt" ]
start_position => beginning
sincedb_path => "/dev/null"
type => "rmlog"
}
}
filter {
if [type] == "rmlog" {
grok {
match => { "message" => "%{HOSTNAME:Hostname},%{DATE:Date},%{HOUR:dt_h}:%{MINUTE:dt_m},%{NUMBER:duration}-%{WORD:hm},%{USER:User},%{USER:User_1} %{NUMBER:Pid} %{NUMBER:float} %{NUMBER:float} %{NUMBER:Num_1} %{NUMBER:Num_2} %{DATA} (?:%{HOUR:dt_h1}:|)(?:%{MINUTE:dt_m1}|) (?:%{HOUR:dt_h2}:|)(?:%{MINUTE:dt_m2}|)%{GREEDYDATA:CMD},%{GREEDYDATA:PWD_PATH}" }
add_field => [ "received_at", "%{@timestamp}" ]
remove_field => [ "@version", "host", "message", "_type", "_index", "_score" ]
}
}
}
output {
if [type] == "rmlog" {
elasticsearch {
hosts => ["myhost:9200"]
manage_template => false
index => "rmlog-%{+YYYY.MM.dd}"
}
}
}