Logstash stops processing files after a while

Centopus · May 20, 2015, 7:28am

Hello

I'm having a problem while trying to parse weblogic log files, via file input plugin.
After a while (took him around 20 minutes to stop reading anything with debug enabled, last time I've tried it survived 5 minutes and before that 7h) it just stops parsing the files. The worst part is, there is no error message, nothing. Just stops working.

I've turned on the debug mode and the result is:

at first is working - finds something... checks other files:

_discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
/TEST/logs/soa_server2/soa_server2.log: file grew, old size 141856, new size 142116 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"96", :method=>"each"}
Received line {:path=>"/TEST/logs/soa_server2/soa_server2.log", :text=>"####<May 20, 2015 8:07:45 AM CEST> <Info> <Health> <GKKT-SOA-TEST2> <soa_server2> <weblogic.GCMonitor> <<anonymous>> <> <7e0c6a05ce1c3e7a:72530d1:14d6fa861fa:-8000-0000000000005968> <1432102065443> <BEA-310002> <67% of the total memory in the server is free> ", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"137", :method=>"run"}
_discover_file_glob: /TEST/logs/AdminServer/TPE-SOA-TEST.log: glob is: ["/TEST/logs/AdminServer/TPE-SOA-TEST.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/*.log: glob is: ["/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/access-201505180200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505140200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505100200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505180200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/access-201505140200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/access-201505100200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505110200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505150200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TEST-FIREWALL-1-access-20150

Then it gradually stops checking the files that change:

_discover_file_glob: /TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/*.log: glob is: ["/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505130200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/tcp-access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505130200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505130200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/tcp-access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505130200.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/logs/AdminServer/TPE-SOA-PROD.log: glob is: ["/PROD/logs/AdminServer/TPE-SOA-PROD.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/logs/AdminServer/TPE-SOA-TEST.log: glob is: ["/TEST/logs/AdminServer/TPE-SOA-TEST.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server.log: file grew, old size 77854, new size 78194 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"96", :method=>"each"}
Received line {:path=>"/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server.log", :text=>"[2015-05-20T08:08:59+02:00] [net-TPE-TEST] [WARNING:1] [OTD-11016] [] [pid: 14106] health-check reports: all servers in origin server pool TPE-BAM-TEST-NSDP are offline.", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"137", :method=>"run"}
_discover_file_glob: /PROD/logs/AdminServer/TPE-SOA-PROD.log: glob is: ["/PROD/logs/AdminServer/TPE-SOA-PROD.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/logs/AdminServer/TPE-SOA-TEST.log: glob is: ["/TEST/logs/AdminServer/TPE-SOA-TEST.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}

Finally reaching to the point, where it checks some files that did not change, and ignores any changes in those:

_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}

At this point logstash process is unable to shutdown properly, must kill it. Any ideas what could be the problem?
I've seen some topics about grok not keeping up, but my grok likes are pretty uncomplicated, I'd say its not this...

//Yes, NMlogs are unprocessed. I know. But that should not be the reason for the anything else to stop.

logstash.conf:

input {

  file {
        type => "WLlog"
        path => ["/TEST/logs/AdminServer/AdminServer.log"]
        tags => [TEST,Admin,log]
       }
  file {
        type => "WLlog"
        path => ["/PROD/logs/AdminServer/AdminServer.log"]
        tags => [PROD,Admin,log]
       }
  file {
        type => "WLlog"
        path => ["/TEST/logs/AdminServer/TEST.log"]
        tags => [TEST,Domain,log]
        }
  file {
        type => "WLlog"
        path => ["/PROD/logs/AdminServer/PROD.log"]
        tags => [PROD,Domain,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/TEST/logs/osb_server1/osb_server1.log",
                        "/TEST/logs/osb_server2/osb_server2.log",
                        "/TEST/logs/osb_server3/osb_server3.log",
                        "/TEST/logs/osb_server4/osb_server4.log",
                        "/TEST/logs/osb_server5/osb_server5.log",
                        "/TEST/logs/osb_server6/osb_server6.log"
                ]
        tags => [TEST,OSB,log]
        }
 file {
        type => "WLlog"
        path => [
                        "/PROD/logs/osb_server1/osb_server1.log",
                        "/PROD/logs/osb_server2/osb_server2.log",
                        "/PROD/logs/osb_server3/osb_server3.log",
                        "/PROD/logs/osb_server4/osb_server4.log",
                        "/PROD/logs/osb_server5/osb_server5.log",
                        "/PROD/logs/osb_server6/osb_server6.log"
                ]
        tags => [PROD,OSB,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/TEST/logs/soa_server1/soa_server1.log",
                        "/TEST/logs/soa_server2/soa_server2.log",
                        "/TEST/logs/soa_server3/soa_server3.log",
                        "/TEST/logs/soa_server4/soa_server4.log",
                        "/TEST/logs/soa_server5/soa_server5.log",
                        "/TEST/logs/soa_server6/soa_server6.log"
                ]
        tags => [TEST,SOA,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/PROD/logs/soa_server1/soa_server1.log",
                        "/PROD/logs/soa_server2/soa_server2.log",
                        "/PROD/logs/soa_server3/soa_server3.log",
                        "/PROD/logs/soa_server4/soa_server4.log",
                        "/PROD/logs/soa_server5/soa_server5.log",
                        "/PROD/logs/soa_server6/soa_server6.log"
                ]
        tags => [PROD,SOA,log]
        }
  file {
        type => "TRlog"
        path => ["/TEST/otdcfg/inst-1/TEST/logs/*.log"]
        tags => [TEST,traf,log,vtd1]
       }
  file {
        type => "TRlog"
        path => ["/TEST/otdcfg/inst-2/TEST/logs/*.log"]
        tags => [TEST,traf,log,vtd2]
       }
  file {
        type => "TRlog"
        path => ["/PROD/otdcfg/inst-1/PROD/logs/*.log"]
        tags => [PROD,traf,log,vtd1]
       }
  file {
        type => "TRlog"
        path => ["/PROD/otdcfg/inst-2/PROD/logs/*.log"]
        tags => [PROD,traf,log,vtd2]
       }
  file {
        type => "NMlog"
        path => [
                        "/TEST/nodemgrs/TEST1/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST2/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST3/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST4/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST5/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST6/logs/nodemanager.log"
                ]
        tags => [TEST,NM,log]
        }
  file {
        type => "NMlog"
        path => [
                        "/PROD/nodemgrs/PROD1/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD2/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD3/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD4/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD5/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD6/logs/nodemanager.log"
                ]
        tags => [PROD,NM,log]
        }
}

filter {
  if [type] == "WLlog"
  {
        multiline {
                   pattern => "^####"
                   negate => true
                   what => "previous"
                   }
        grok {
             match => [ "message", "####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{HOST:hostname}> <(%{WORD:server})?> %{GREEDYDATA:logmessage}"]
             }
  }
  if [type] == "TRlog"
                {
                grok    {
                        match => [ "message", "%{IP:sourceIP} - - \[%{DATA:timestamp}\] \"%{DATA:action}\" %{NUMBER:code} %{NUMBER:size} %{DATA:messageID} %{HOST:hostname}:%{NUMBER:port}"]
                        }
                }
}

output { elasticsearch { host => "elastic_machine"
                         cluster => FreeWorldAlliance
                       }
               }

Centopus · May 20, 2015, 10:06am

When i try to kill it, it shows this:

_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4faa208f sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5554f7f9 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x2156f7c9 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0xe21f971 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6bb19e77 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x239e6bfa sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5e772c0 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6ff89d06 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x43e1f00 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x584f877a sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x46cc6d2d sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x52a10821 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3608e5ed sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3d1cd145 sleep>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"}
SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4faa208f dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5554f7f9 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x2156f7c9 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0xe21f971 run>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6bb19e77 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x239e6bfa run>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5e772c0 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6ff89d06 run>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x43e1f00 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x584f877a dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x46cc6d2d dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x52a10821 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3608e5ed dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3d1cd145 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"}

PS. I'm having some problems with posting here, sorry for any post duplicates.

Centopus · May 20, 2015, 10:53am

Cant kill it, but sending a shutdown request from elasticsearch (using kopf), made it shutdown.

SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4faa208f dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5554f7f9 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x2156f7c9 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0xe21f971 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6bb19e77 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x239e6bfa dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x5e772c0 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6ff89d06 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x43e1f00 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x584f877a dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x46cc6d2d dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x52a10821 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3608e5ed dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3d1cd145 dead>, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"}
May 20, 2015 12:51:18 PM org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$NodeShutdownRequestHandler messageReceived
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] shutting down in [200ms]
May 20, 2015 12:51:19 PM org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$NodeShutdownRequestHandler$1 run
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] initiating requested shutdown...
May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode stop
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] stopping ...
May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode stop
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] stopped
May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode close
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] closing ...
May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode close
INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] closed

spuder · May 20, 2015, 6:29pm

I've seen the same issue, and it appears that others have encountered the same problem.
Try reinstalling your plugins
The frequency has been reduced since I upgraded to 1.5rc4

Logstash can not be restarted

opened 05:31PM - 13 Apr 15 UTC

closed 04:28PM - 11 May 15 UTC

spuder

bug

I've encountered this problem 3 times now over the course of a month. A brand …new install of logstash 1.5rc2 on ubuntu 14.04 will get into a state where I can not stop or restart the process. (Installed with chef using community cookbooks). When attempting to restart logstash, it timesout and says 'got TERM' ``` service logstash_server restart timeout: run: logstash_server: (pid 11797) 839785s, got TERM ``` Also asking stackoverflow for help in identifying the problem http://unix.stackexchange.com/questions/195998/how-to-identify-why-a-process-wont-die

github.com/elastic/logstash

Logstash 1.5 crashes if it encounters communication problem

opened 03:13PM - 20 Apr 15 UTC

closed 05:56AM - 18 May 15 UTC

spuder

bug

This issue is related to #2992 If logstash 1.5rc2 encounters a problem writing… to elasticsearch, it will cause logtsash to crash. Full logs are shown here: http://pastebin.com/tdy8KWay The interesting lines are: ``` {:timestamp=>"2015-04-17T11:34:24.192000-0600", :message=>"Got error to send bulk of actions to elasticsearch server at swat-elasticsearchpool.ndlab.local : Read timed out", :level=>:error} {:timestamp=>"2015-04-17T11:34:24.193000-0600", :message=>"Failed to flush outgoing items", :outgoing_count=>5000, :exception=>#<Manticore::Timeout: Read timed out>, :backtrace=>["/opt/logstash/server/vendor/bundle/jruby/1.9/gems/manticore-0.3.5-java/lib/manticore/response.rb:35:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/manticore-0.3.5-java/lib/manticore/response.rb:61:in `call'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/manticore-0.3.5-java/lib/manticore/response.rb:224:in `call_once'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/manticore-0.3.5-java/lib/manticore/response.rb:127:in `code'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.7/lib/elasticsearch/transport/transport/http/manticore.rb:50:in `perform_request'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.7/lib/elasticsearch/transport/transport/base.rb:187:in `perform_request'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.7/lib/elasticsearch/transport/transport/http/manticore.rb:33:in `perform_request'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.7/lib/elasticsearch/transport/client.rb:115:in `perform_request'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.7/lib/elasticsearch/api/actions/bulk.rb:80:in `bulk'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch/protocol.rb:82:in `bulk'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:413:in `submit'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:412:in `submit'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:438:in `flush'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:436:in `flush'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1341:in `each'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:216:in `buffer_flush'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:193:in `buffer_flush'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:159:in `buffer_receive'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:402:in `receive'", "/opt/logstash/server/lib/logstash/outputs/base.rb:88:in `handle'", "(eval):233:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/server/lib/logstash/pipeline.rb:279:in `output'", "/opt/logstash/server/lib/logstash/pipeline.rb:235:in `outputworker'", "/opt/logstash/server/lib/logstash/pipeline.rb:163:in `start_outputs'"], :level=>:warn} {:timestamp=>"2015-04-17T14:09:04.890000-0600", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn} ``` Because of issue #2992 the logstash service will fail to be restarted by the upstart watchdog on ubuntu 14.04. The config files are: ``` input { kafka { zk_connect => 'swat-zoo05.example.com:2181/kafka' consumer_threads => 3 topic_id => 'foobar' } } filter { mutate { gsub => [ "[id][batchID]", "\D", "" ] } mutate { convert => [ "[id][batchID]", "integer"] } } filter { mutate { gsub => [ "[id][docID]", "\D", "" ] } mutate { convert => [ "[id][docID]", "integer"] } } output { elasticsearch { host => 'swat-elasticsearchpool.example.com' cluster => 'foobar-elastic' embedded => false protocol => 'http' } } ```

Centopus · May 21, 2015, 10:45am

Thank you.
I've reinstalled my plugins and added an additional CPU to the machine (I've noticed it was running on 100% at times) it may have contributed too.
I'll report back after it was running for a while or crashed again.

Centopus · May 22, 2015, 5:22am

Naaah It died after 2h and on second try after 30 minutes...

Now it does not saturate the cpu, so that was not the problem for sure.

I think I'm going to have to wait for the next version and fallback to 1.4 for now.

Found something:

Going to try updating ES to newer one (currently running 1.5.1).

And this:

And this one seems to be exacly same issue as mine:

-----------------------------------------------------------------Updating now works for me, no new posts unless someone responds --------------------------------------------

root     19920 29.4  5.7 2350544 468308 pts/0  Sl+  08:05   5:25  |                       \_ /logstash/jdk1.8.0_45/jre/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xmx500m -Xss2048k -Djffi.boot.library.path=/logstash/logstash-1.5.0/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xbootclasspath/a:/logstash/logstash-1.5.0/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/logstash/logstash-1.5.0/vendor/jruby -Djruby.lib=/logstash/logstash-1.5.0/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /logstash/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f ../conf/logstash.conf -v

And then strace:

[root@somwhere]# strace -p 19920
Process 19920 attached - interrupt to quit
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL
 <unfinished ...>
Process 19920 detached
[root@somwhere]# strace -p 19920
Process 19920 attached - interrupt to quit
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---
futex(0x7f098faebf00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn(0x7f098faebf00)            = 202
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---
futex(0x7f098faebf00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn(0x7f098faebf00)            = 202
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---
futex(0x7f098faebf00, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn(0x7f098faebf00)            = 202
futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULLPANIC: attached pid 19920 exited with 0
 <unfinished ... exit status 0>

It finally died. (I'm running it from root, because I'm going to add syslog input when its stable; from embedded systems, no port change option).