Hi there, -Hope this is the right place to put my problem-
I am using Logstash 6.0.0 to parse logs from Filebeat 6.0.0 instances. I tried to change some IP addresses with server names using the DNS filter but after a while processing (less than a day), Logstash stops writing logs and sending messages to ElasticSearch (also 6.0.0). But even without -apparently- doing anything, Logstash is still running and need a kill -9 to stop...
I have two logstash instances running on RedHat7 servers with Xms and Xmx configured to 8g.
I assume that my problem is related to the DNS filter because when I comment DNS configuration lines, everything works fine.
Here are the parts of my logstash configuration using DNS filter :
grok {
match => { "message" => [
"^%{GREEDYDATA:clientip}, %{IPORHOST:ident} - - \[%{HTTPDATE:log_date}\] %{NUMBER:response_time:int} %{QS:request} %{NUMBER:response:int} (?:%{NUMBER:bytes:int}|-) (%{QS:referrer}|-) %{QS:agent}$",
"^%{GREEDYDATA:clientip} - - \[%{HTTPDATE:log_date}\] %{NUMBER:response_time:int} %{QS:request} %{NUMBER:response:int} (?:%{NUMBER:bytes:int}|-) (%{QS:referrer}|-) %{QS:agent}$"
] }
tag_on_failure => ["_apacheparsingfailure"]
}
# dns {
# reverse => [ "clientip" ]
# action => "replace"
# }
And in the exact same way for other logs :
# dns {
# reverse => [ "hostaddr" ]
# action => [ "replace" ]
# }
We have a old version (Logstash 1.5.2) of the stack running (Which I did not work on) that does not have this problem.
I was wondering if it could be a DNS server problem because I receive avout 10 000 messages/5min that pass in one of the two filters here, but I do not know if it could be too much or not. (The working old version had about 4 000 messages/5min)
Those part of my configuration are copy-pasting of what was used with Logstash 1.5.2 (before the decision to restart the whole project with new versions of tools and new servers) so I might have missed a breaking change or something.
Before asking here I made some searches and found some similar problems, not necessarily related to dns, but couldn't resolve mine...
Here are some of those links :
https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944/3
https://github.com/logstash-plugins/logstash-input-lumberjack/issues/9
https://github.com/elastic/logstash/issues/3914
I think that if there really was a bug somewhere someone would have hit it before me, so I hope that someone here knows this issue or understand it and could give me a workaround
Any help would be VERY appreciated