Logstash stops processing files after a while

[Centopus]

Hello

I'm having a problem while trying to parse weblogic log files, via file input plugin.
After a while (took him around 20 minutes to stop reading anything with debug enabled, last time I've tried it survived 5 minutes and before that 7h) it just stops parsing the files. The worst part is, there is no error message, nothing. Just stops working.

I've turned on the debug mode and the result is:

at first is working - finds something... checks other files:

_discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
/TEST/logs/soa_server2/soa_server2.log: file grew, old size 141856, new size 142116 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"96", :method=>"each"}
Received line {:path=>"/TEST/logs/soa_server2/soa_server2.log", :text=>"####<May 20, 2015 8:07:45 AM CEST> <Info> <Health> <GKKT-SOA-TEST2> <soa_server2> <weblogic.GCMonitor> <<anonymous>> <> <7e0c6a05ce1c3e7a:72530d1:14d6fa861fa:-8000-0000000000005968> <1432102065443> <BEA-310002> <67% of the total memory in the server is free> ", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"137", :method=>"run"}
_discover_file_glob: /TEST/logs/AdminServer/TPE-SOA-TEST.log: glob is: ["/TEST/logs/AdminServer/TPE-SOA-TEST.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/*.log: glob is: ["/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/access-201505180200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505140200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505100200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505180200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/access-201505140200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/access-201505100200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505110200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505150200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TEST-FIREWALL-1-access-20150

Then it gradually stops checking the files that change:

_discover_file_glob: /TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/*.log: glob is: ["/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505130200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/tcp-access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505130200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505130200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/tcp-access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505130200.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/logs/AdminServer/TPE-SOA-PROD.log: glob is: ["/PROD/logs/AdminServer/TPE-SOA-PROD.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/logs/AdminServer/TPE-SOA-TEST.log: glob is: ["/TEST/logs/AdminServer/TPE-SOA-TEST.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server.log: file grew, old size 77854, new size 78194 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"96", :method=>"each"}
Received line {:path=>"/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server.log", :text=>"[2015-05-20T08:08:59+02:00] [net-TPE-TEST] [WARNING:1] [OTD-11016] [] [pid: 14106] health-check reports: all servers in origin server pool TPE-BAM-TEST-NSDP are offline.", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"137", :method=>"run"}
_discover_file_glob: /PROD/logs/AdminServer/TPE-SOA-PROD.log: glob is: ["/PROD/logs/AdminServer/TPE-SOA-PROD.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/logs/AdminServer/TPE-SOA-TEST.log: glob is: ["/TEST/logs/AdminServer/TPE-SOA-TEST.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}

Finally reaching to the point, where it checks some files that did not change, and ignores any changes in those:

_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}

At this point logstash process is unable to shutdown properly, must kill it. Any ideas what could be the problem?
I've seen some topics about grok not keeping up, but my grok likes are pretty uncomplicated, I'd say its not this...

//Yes, NMlogs are unprocessed. I know. But that should not be the reason for the anything else to stop.

logstash.conf:

input {

  file {
        type => "WLlog"
        path => ["/TEST/logs/AdminServer/AdminServer.log"]
        tags => [TEST,Admin,log]
       }
  file {
        type => "WLlog"
        path => ["/PROD/logs/AdminServer/AdminServer.log"]
        tags => [PROD,Admin,log]
       }
  file {
        type => "WLlog"
        path => ["/TEST/logs/AdminServer/TEST.log"]
        tags => [TEST,Domain,log]
        }
  file {
        type => "WLlog"
        path => ["/PROD/logs/AdminServer/PROD.log"]
        tags => [PROD,Domain,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/TEST/logs/osb_server1/osb_server1.log",
                        "/TEST/logs/osb_server2/osb_server2.log",
                        "/TEST/logs/osb_server3/osb_server3.log",
                        "/TEST/logs/osb_server4/osb_server4.log",
                        "/TEST/logs/osb_server5/osb_server5.log",
                        "/TEST/logs/osb_server6/osb_server6.log"
                ]
        tags => [TEST,OSB,log]
        }
 file {
        type => "WLlog"
        path => [
                        "/PROD/logs/osb_server1/osb_server1.log",
                        "/PROD/logs/osb_server2/osb_server2.log",
                        "/PROD/logs/osb_server3/osb_server3.log",
                        "/PROD/logs/osb_server4/osb_server4.log",
                        "/PROD/logs/osb_server5/osb_server5.log",
                        "/PROD/logs/osb_server6/osb_server6.log"
                ]
        tags => [PROD,OSB,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/TEST/logs/soa_server1/soa_server1.log",
                        "/TEST/logs/soa_server2/soa_server2.log",
                        "/TEST/logs/soa_server3/soa_server3.log",
                        "/TEST/logs/soa_server4/soa_server4.log",
                        "/TEST/logs/soa_server5/soa_server5.log",
                        "/TEST/logs/soa_server6/soa_server6.log"
                ]
        tags => [TEST,SOA,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/PROD/logs/soa_server1/soa_server1.log",
                        "/PROD/logs/soa_server2/soa_server2.log",
                        "/PROD/logs/soa_server3/soa_server3.log",
                        "/PROD/logs/soa_server4/soa_server4.log",
                        "/PROD/logs/soa_server5/soa_server5.log",
                        "/PROD/logs/soa_server6/soa_server6.log"
                ]
        tags => [PROD,SOA,log]
        }
  file {
        type => "TRlog"
        path => ["/TEST/otdcfg/inst-1/TEST/logs/*.log"]
        tags => [TEST,traf,log,vtd1]
       }
  file {
        type => "TRlog"
        path => ["/TEST/otdcfg/inst-2/TEST/logs/*.log"]
        tags => [TEST,traf,log,vtd2]
       }
  file {
        type => "TRlog"
        path => ["/PROD/otdcfg/inst-1/PROD/logs/*.log"]
        tags => [PROD,traf,log,vtd1]
       }
  file {
        type => "TRlog"
        path => ["/PROD/otdcfg/inst-2/PROD/logs/*.log"]
        tags => [PROD,traf,log,vtd2]
       }
  file {
        type => "NMlog"
        path => [
                        "/TEST/nodemgrs/TEST1/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST2/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST3/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST4/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST5/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST6/logs/nodemanager.log"
                ]
        tags => [TEST,NM,log]
        }
  file {
        type => "NMlog"
        path => [
                        "/PROD/nodemgrs/PROD1/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD2/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD3/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD4/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD5/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD6/logs/nodemanager.log"
                ]
        tags => [PROD,NM,log]
        }
}

filter {
  if [type] == "WLlog"
  {
        multiline {
                   pattern => "^####"
                   negate => true
                   what => "previous"
                   }
        grok {
             match => [ "message", "####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{HOST:hostname}> <(%{WORD:server})?> %{GREEDYDATA:logmessage}"]
             }
  }
  if [type] == "TRlog"
                {
                grok    {
                        match => [ "message", "%{IP:sourceIP} - - \[%{DATA:timestamp}\] \"%{DATA:action}\" %{NUMBER:code} %{NUMBER:size} %{DATA:messageID} %{HOST:hostname}:%{NUMBER:port}"]
                        }
                }
}

output { elasticsearch { host => "elastic_machine"
                         cluster => FreeWorldAlliance
                       }
               }

[Centopus]

When i try to kill it, it shows this:

_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4faa208f sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5554f7f9 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x2156f7c9 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0xe21f971 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6bb19e77 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x239e6bfa sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5e772c0 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6ff89d06 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x43e1f00 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x584f877a sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x46cc6d2d sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x52a10821 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3608e5ed sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3d1cd145 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4faa208f dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5554f7f9 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x2156f7c9 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0xe21f971 run>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6bb19e77 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x239e6bfa run>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5e772c0 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6ff89d06 run>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x43e1f00 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x584f877a dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x46cc6d2d dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x52a10821 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3608e5ed dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3d1cd145 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"}

PS. I'm having some problems with posting here, sorry for any post duplicates.

[Centopus]

Cant kill it, but sending a shutdown request from elasticsearch (using kopf), made it shutdown.

SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4faa208f dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5554f7f9 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x2156f7c9 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0xe21f971 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6bb19e77 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x239e6bfa dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5e772c0 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6ff89d06 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x43e1f00 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x584f877a dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x46cc6d2d dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x52a10821 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3608e5ed dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3d1cd145 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
May 20, 2015 12:51:18 PM org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$NodeShutdownRequestHandler messageReceived
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] shutting down in [200ms]
May 20, 2015 12:51:19 PM org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$NodeShutdownRequestHandler$1 run
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] initiating requested shutdown...
May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode stop
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] stopping ...
May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode stop
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] stopped
May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode close
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] closing ...
May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode close
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] closed

[spuder]

I've seen the same issue, and it appears that others have encountered the same problem.
Try reinstalling your plugins
The frequency has been reduced since I upgraded to 1.5rc4

Related:

Logstash can not be restarted

Why won't this process die after SIGTERM

1.5.0 works from CLI, but not from Puppet elasticsearc/logstash module

I've been encountering the same problem with logstash 1.5rc4 and kafka 0.8.2. I installed logstash from the chef cookbook. For 2 days, logstash would not read any data from kafka when started as a service, but when I started it from the command line, the data would stream in correctly. I rebooted my kafka servers, and the problem appears to have mostly gone away. I suspect it was a zookeeper related problem. http://stackoverflow.com/questions/29276912/kafka-suddenly-reset-the-consumer-offset#…

Logstash 1.5 crashes if it encounters communication problem

[Centopus]

Thank you.
I've reinstalled my plugins and added an additional CPU to the machine (I've noticed it was running on 100% at times) it may have contributed too.
I'll report back after it was running for a while or crashed again.

[Centopus]

Naaah It died after 2h and on second try after 30 minutes...

Now it does not saturate the cpu, so that was not the problem for sure.

I think I'm going to have to wait for the next version and fallback to 1.4 for now.

---

Found something:

logstash 1.5 RC2 stops after a while processing...

Going to try updating ES to newer one (currently running 1.5.1).

And this:

Logstash Stalls - logstash-1.5.0_rc3-1

And this one seems to be exacly same issue as mine:

Logstash 1:1.5.0-1 dies after some time

-----------------------------------------------------------------Updating now works for me, no new posts unless someone responds --------------------------------------------

root     19920 29.4  5.7 2350544 468308 pts/0  Sl+  08:05   5:25  |                       \_ /logstash/jdk1.8.0_45/jre/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xmx500m -Xss2048k -Djffi.boot.library.path=/logstash/logstash-1.5.0/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xbootclasspath/a:/logstash/logstash-1.5.0/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/logstash/logstash-1.5.0/vendor/jruby -Djruby.lib=/logstash/logstash-1.5.0/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /logstash/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f ../conf/logstash.conf -v

And then strace:

[root@somwhere]# strace -p 19920
Process 19920 attached - interrupt to quit
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL
 <unfinished ...>
Process 19920 detached
[root@somwhere]# strace -p 19920
Process 19920 attached - interrupt to quit
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---
futex(0x7f098faebf00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn(0x7f098faebf00)            = 202
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---
futex(0x7f098faebf00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn(0x7f098faebf00)            = 202
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---
futex(0x7f098faebf00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn(0x7f098faebf00)            = 202
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULLPANIC: attached pid 19920 exited with 0
 <unfinished ... exit status 0>

It finally died. (I'm running it from root, because I'm going to add syslog input when its stable; from embedded systems, no port change option).