# Logstash stops processing files after a while **URL:** \ **Category:** Logstash\ **Created:** [May 20, 2015, 7:28am UTC](https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944 "2015-05-20T07:28:46Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Centopus](https://avatars.discourse-cdn.com/v4/letter/c/838e76/32.png) [@Centopus](https://discuss.elastic.co/u/Centopus)\ **Post date:** [May 20, 2015, 7:28am UTC](https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944/1 "2015-05-20T07:28:46Z")
Hello I'm having a problem while trying to parse weblogic log files, via file input plugin. After a while (took him around 20 minutes to stop reading anything with debug enabled, last time I've tried it survived 5 minutes and before that 7h) it just stops parsing the files. The worst part is, there is no error message, nothing. Just stops working. I've turned on the debug mode and the result is: at first is working - finds something... checks other files: ``` _discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} /TEST/logs/soa_server2/soa_server2.log: file grew, old size 141856, new size 142116 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"96", :method=>"each"} Received line {:path=>"/TEST/logs/soa_server2/soa_server2.log", :text=>"#### <> <> <7e0c6a05ce1c3e7a:72530d1:14d6fa861fa:-8000-0000000000005968> <1432102065443> <67% of the total memory in the server is free> ", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"137", :method=>"run"} _discover_file_glob: /TEST/logs/AdminServer/TPE-SOA-TEST.log: glob is: ["/TEST/logs/AdminServer/TPE-SOA-TEST.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/*.log: glob is: ["/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/access-201505180200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505140200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505100200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505180200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/access-201505140200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/access-201505100200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505110200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505150200.log", "/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TEST-FIREWALL-1-access-20150 ``` Then it gradually stops checking the files that change: ``` _discover_file_glob: /TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/*.log: glob is: ["/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505130200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/tcp-access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505130200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505130200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/tcp-access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505020200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505150200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504250200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504210200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505110200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505060200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504290200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505190200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505090200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504280200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505050200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504220200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505070200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201504260200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505030200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201504240200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505010200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505200200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505100200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505120200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server-201505160200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505140200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TPE-OSB-TEST-access-201505180200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505080200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505170200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504270200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201504300200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201504230200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/TEST-FIREWALL-1-access-201505040200.log", "/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/access-201505130200.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/logs/AdminServer/TPE-SOA-PROD.log: glob is: ["/PROD/logs/AdminServer/TPE-SOA-PROD.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/logs/AdminServer/TPE-SOA-TEST.log: glob is: ["/TEST/logs/AdminServer/TPE-SOA-TEST.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} /TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server.log: file grew, old size 77854, new size 78194 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"96", :method=>"each"} Received line {:path=>"/TEST/otdcfg/otd-inst-2/net-TPE-TEST/logs/server.log", :text=>"[2015-05-20T08:08:59+02:00] [net-TPE-TEST] [WARNING:1] [OTD-11016] [] [pid: 14106] health-check reports: all servers in origin server pool TPE-BAM-TEST-NSDP are offline.", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"137", :method=>"run"} _discover_file_glob: /PROD/logs/AdminServer/TPE-SOA-PROD.log: glob is: ["/PROD/logs/AdminServer/TPE-SOA-PROD.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/logs/AdminServer/TPE-SOA-TEST.log: glob is: ["/TEST/logs/AdminServer/TPE-SOA-TEST.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST1/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST2/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST3/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST4/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST5/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log: glob is: ["/TEST/nodemgrs/GKKT-SOA-TEST6/logs/nodemanager.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} ``` Finally reaching to the point, where it checks some files that did not change, and ignores any changes in those: ``` _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD4/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD5/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD2/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD3/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} ``` At this point logstash process is unable to shutdown properly, must kill it. Any ideas what could be the problem? I've seen some topics about grok not keeping up, but my grok likes are pretty uncomplicated, I'd say its not this... //Yes, NMlogs are unprocessed. I know. But that should not be the reason for the anything else to stop. logstash.conf: ``` input { file { type => "WLlog" path => ["/TEST/logs/AdminServer/AdminServer.log"] tags => [TEST,Admin,log] } file { type => "WLlog" path => ["/PROD/logs/AdminServer/AdminServer.log"] tags => [PROD,Admin,log] } file { type => "WLlog" path => ["/TEST/logs/AdminServer/TEST.log"] tags => [TEST,Domain,log] } file { type => "WLlog" path => ["/PROD/logs/AdminServer/PROD.log"] tags => [PROD,Domain,log] } file { type => "WLlog" path => [ "/TEST/logs/osb_server1/osb_server1.log", "/TEST/logs/osb_server2/osb_server2.log", "/TEST/logs/osb_server3/osb_server3.log", "/TEST/logs/osb_server4/osb_server4.log", "/TEST/logs/osb_server5/osb_server5.log", "/TEST/logs/osb_server6/osb_server6.log" ] tags => [TEST,OSB,log] } file { type => "WLlog" path => [ "/PROD/logs/osb_server1/osb_server1.log", "/PROD/logs/osb_server2/osb_server2.log", "/PROD/logs/osb_server3/osb_server3.log", "/PROD/logs/osb_server4/osb_server4.log", "/PROD/logs/osb_server5/osb_server5.log", "/PROD/logs/osb_server6/osb_server6.log" ] tags => [PROD,OSB,log] } file { type => "WLlog" path => [ "/TEST/logs/soa_server1/soa_server1.log", "/TEST/logs/soa_server2/soa_server2.log", "/TEST/logs/soa_server3/soa_server3.log", "/TEST/logs/soa_server4/soa_server4.log", "/TEST/logs/soa_server5/soa_server5.log", "/TEST/logs/soa_server6/soa_server6.log" ] tags => [TEST,SOA,log] } file { type => "WLlog" path => [ "/PROD/logs/soa_server1/soa_server1.log", "/PROD/logs/soa_server2/soa_server2.log", "/PROD/logs/soa_server3/soa_server3.log", "/PROD/logs/soa_server4/soa_server4.log", "/PROD/logs/soa_server5/soa_server5.log", "/PROD/logs/soa_server6/soa_server6.log" ] tags => [PROD,SOA,log] } file { type => "TRlog" path => ["/TEST/otdcfg/inst-1/TEST/logs/*.log"] tags => [TEST,traf,log,vtd1] } file { type => "TRlog" path => ["/TEST/otdcfg/inst-2/TEST/logs/*.log"] tags => [TEST,traf,log,vtd2] } file { type => "TRlog" path => ["/PROD/otdcfg/inst-1/PROD/logs/*.log"] tags => [PROD,traf,log,vtd1] } file { type => "TRlog" path => ["/PROD/otdcfg/inst-2/PROD/logs/*.log"] tags => [PROD,traf,log,vtd2] } file { type => "NMlog" path => [ "/TEST/nodemgrs/TEST1/logs/nodemanager.log", "/TEST/nodemgrs/TEST2/logs/nodemanager.log", "/TEST/nodemgrs/TEST3/logs/nodemanager.log", "/TEST/nodemgrs/TEST4/logs/nodemanager.log", "/TEST/nodemgrs/TEST5/logs/nodemanager.log", "/TEST/nodemgrs/TEST6/logs/nodemanager.log" ] tags => [TEST,NM,log] } file { type => "NMlog" path => [ "/PROD/nodemgrs/PROD1/logs/nodemanager.log", "/PROD/nodemgrs/PROD2/logs/nodemanager.log", "/PROD/nodemgrs/PROD3/logs/nodemanager.log", "/PROD/nodemgrs/PROD4/logs/nodemanager.log", "/PROD/nodemgrs/PROD5/logs/nodemanager.log", "/PROD/nodemgrs/PROD6/logs/nodemanager.log" ] tags => [PROD,NM,log] } } filter { if [type] == "WLlog" { multiline { pattern => "^####" negate => true what => "previous" } grok { match => ["message", "####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{HOST:hostname}> <(%{WORD:server})?> %{GREEDYDATA:logmessage}"] } } if [type] == "TRlog" { grok { match => ["message", "%{IP:sourceIP} - - \[%{DATA:timestamp}\] \"%{DATA:action}\" %{NUMBER:code} %{NUMBER:size} %{DATA:messageID} %{HOST:hostname}:%{NUMBER:port}"] } } } output { elasticsearch { host => "elastic_machine" cluster => FreeWorldAlliance } } ``` ---
**Author:** ![Centopus](https://avatars.discourse-cdn.com/v4/letter/c/838e76/32.png) [@Centopus](https://discuss.elastic.co/u/Centopus)\ **Post date:** [May 20, 2015, 10:06am UTC](https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944/2 "2015-05-20T10:06:41Z")
When i try to kill it, it shows this: ``` _discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD6/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"} SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} caller requested sincedb write () {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"202", :method=>"sincedb_write"} SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"} ``` PS. I'm having some problems with posting here, sorry for any post duplicates. ---
**Author:** ![Centopus](https://avatars.discourse-cdn.com/v4/letter/c/838e76/32.png) [@Centopus](https://discuss.elastic.co/u/Centopus)\ **Post date:** [May 20, 2015, 10:53am UTC](https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944/3 "2015-05-20T10:53:05Z")
Cant kill it, but sending a shutdown request from elasticsearch (using kopf), made it shutdown. ``` SIGINT received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"116", :method=>"execute"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} Sending shutdown signal to input thread {:thread=>#, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"258", :method=>"shutdown"} May 20, 2015 12:51:18 PM org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$NodeShutdownRequestHandler messageReceived INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] shutting down in [200ms] May 20, 2015 12:51:19 PM org.elasticsearch.action.admin.cluster.node.shutdown.TransportNodesShutdownAction$NodeShutdownRequestHandler$1 run INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] initiating requested shutdown... May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode stop INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] stopping ... May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode stop INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] stopped May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode close INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] closing ... May 20, 2015 12:51:19 PM org.elasticsearch.node.internal.InternalNode close INFO: [logstash-GKKT-SOA-BORG1-10-170-13-20-20430-12728] closed ``` ---
**Author:** ![spuder](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spuder/32/44895_2.png) [@spuder](https://discuss.elastic.co/u/spuder)\ **Post date:** [May 20, 2015, 6:29pm UTC](https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944/4 "2015-05-20T18:29:44Z")
I've seen the same issue, and it appears that others have encountered the same problem. Try reinstalling your plugins The frequency has been reduced since I upgraded to 1.5rc4 Related: > > > I've encountered this problem 3 times now over the course of a month. > > A brand …new install of logstash 1.5rc2 on ubuntu 14.04 will get into a state where I can not stop or restart the process. (Installed with chef using community cookbooks). > > When attempting to restart logstash, it timesout and says 'got TERM' > > \`\`\` > service logstash\_server restart > timeout: run: logstash\_server: (pid 11797) 839785s, got TERM > \`\`\` > > Also asking stackoverflow for help in identifying the problem > > http://unix.stackexchange.com/questions/195998/how-to-identify-why-a-process-wont-die > > [@1.5.0 works from CLI, but not from Puppet elasticsearc/logstash module](https://discuss.elastic.co/t/1-5-0-works-from-cli-but-not-from-puppet-elasticsearc-logstash-module/873/3): > > I've been encountering the same problem with logstash 1.5rc4 and kafka 0.8.2. I installed logstash from the chef cookbook. For 2 days, logstash would not read any data from kafka when started as a service, but when I started it from the command line, the data would stream in correctly. I rebooted my kafka servers, and the problem appears to have mostly gone away. I suspect it was a zookeeper related problem. [http://stackoverflow.com/questions/29276912/kafka-suddenly-reset-the-consumer-offset#…](http://stackoverflow.com/questions/29276912/kafka-suddenly-reset-the-consumer-offset#comment48755603_29276912) > > > This issue is related to #2992 > > If logstash 1.5rc2 encounters a problem writing… to elasticsearch, it will cause logtsash to crash. > > Full logs are shown here: > > http://pastebin.com/tdy8KWay > > The interesting lines are: > > \`\`\` > {:timestamp=\>"2015-04-17T11:34:24.192000-0600", :message=\>"Got error to send bulk of actions to elasticsearch server at swat-elasticsearchpool.ndlab.local : Read timed out", :level=\>:error} > {:timestamp=\>"2015-04-17T11:34:24.193000-0600", :message=\>"Failed to flush outgoing items", :outgoing\_count=\>5000, :exception=\>#\, :backtrace=\>\["/opt/logstash/server/vendor/bundle/jruby/1.9/gems/manticore-0.3.5-java/lib/manticore/response.rb:35:in \`initialize'", "org/jruby/RubyProc.java:271:in \`call'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/manticore-0.3.5-java/lib/manticore/response.rb:61:in \`call'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/manticore-0.3.5-java/lib/manticore/response.rb:224:in \`call\_once'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/manticore-0.3.5-java/lib/manticore/response.rb:127:in \`code'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.7/lib/elasticsearch/transport/transport/http/manticore.rb:50:in \`perform\_request'", "org/jruby/RubyProc.java:271:in \`call'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.7/lib/elasticsearch/transport/transport/base.rb:187:in \`perform\_request'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.7/lib/elasticsearch/transport/transport/http/manticore.rb:33:in \`perform\_request'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.7/lib/elasticsearch/transport/client.rb:115:in \`perform\_request'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.7/lib/elasticsearch/api/actions/bulk.rb:80:in \`bulk'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch/protocol.rb:82:in \`bulk'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:413:in \`submit'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:412:in \`submit'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:438:in \`flush'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:436:in \`flush'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:219:in \`buffer\_flush'", "org/jruby/RubyHash.java:1341:in \`each'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:216:in \`buffer\_flush'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:193:in \`buffer\_flush'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:159:in \`buffer\_receive'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.1.18-java/lib/logstash/outputs/elasticsearch.rb:402:in \`receive'", "/opt/logstash/server/lib/logstash/outputs/base.rb:88:in \`handle'", "(eval):233:in \`initialize'", "org/jruby/RubyProc.java:271:in \`call'", "/opt/logstash/server/lib/logstash/pipeline.rb:279:in \`output'", "/opt/logstash/server/lib/logstash/pipeline.rb:235:in \`outputworker'", "/opt/logstash/server/lib/logstash/pipeline.rb:163:in \`start\_outputs'"\], :level=\>:warn} > {:timestamp=\>"2015-04-17T14:09:04.890000-0600", :message=\>"SIGTERM received. Shutting down the pipeline.", :level=\>:warn} > \`\`\` > > Because of issue #2992 the logstash service will fail to be restarted by the upstart watchdog on ubuntu 14.04. > > The config files are: > > \`\`\` > input { > kafka { > zk\_connect =\> 'swat-zoo05.example.com:2181/kafka' > consumer\_threads =\> 3 > topic\_id =\> 'foobar' > } > } > filter { > mutate { > gsub =\> \["\[id\]\[batchID\]", "\\D", "" \] > } > mutate { > convert =\> \["\[id\]\[batchID\]", "integer"\] > } > } > filter { > mutate { > gsub =\> \["\[id\]\[docID\]", "\\D", "" \] > } > mutate { > convert =\> \["\[id\]\[docID\]", "integer"\] > } > } > output { > elasticsearch { > host =\> 'swat-elasticsearchpool.example.com' > cluster =\> 'foobar-elastic' > embedded =\> false > protocol =\> 'http' > } > } > \`\`\` ---
**Author:** ![Centopus](https://avatars.discourse-cdn.com/v4/letter/c/838e76/32.png) [@Centopus](https://discuss.elastic.co/u/Centopus)\ **Post date:** [May 21, 2015, 10:45am UTC](https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944/5 "2015-05-21T10:45:10Z")
Thank you. I've reinstalled my plugins and added an additional CPU to the machine (I've noticed it was running on 100% at times) it may have contributed too. I'll report back after it was running for a while or crashed again. ---
**Author:** ![Centopus](https://avatars.discourse-cdn.com/v4/letter/c/838e76/32.png) [@Centopus](https://discuss.elastic.co/u/Centopus)\ **Post date:** [May 22, 2015, 5:22am UTC](https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944/6 "2015-05-22T05:22:49Z")
Naaah It died after 2h and on second try after 30 minutes... Now it does not saturate the cpu, so that was not the problem for sure. I think I'm going to have to wait for the next version and fallback to 1.4 for now. * * * Found something: > > > We have about 300 events per second and while 1.4.2 was working perfectly for weeks 1.5 RC2 stops after an unpredictable... Going to try updating ES to newer one (currently running 1.5.1). And this: > > > Hi, > I've noticed after upgrading to RC3-1 that logstash hangs/freezes after a few seconds of starting. Just using the netflow codec. > OS: Centos... And this one seems to be exacly same issue as mine: > > > Hi, today I've upgraded to logstash version 1:1.5.0-1 on ubuntu 14.04, x86\_64, and it seems to hangs after some time without... -----------------------------------------------------------------Updating now works for me, no new posts unless someone responds -------------------------------------------- ``` root 19920 29.4 5.7 2350544 468308 pts/0 Sl+ 08:05 5:25 | \_ /logstash/jdk1.8.0_45/jre/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xmx500m -Xss2048k -Djffi.boot.library.path=/logstash/logstash-1.5.0/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xbootclasspath/a:/logstash/logstash-1.5.0/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/logstash/logstash-1.5.0/vendor/jruby -Djruby.lib=/logstash/logstash-1.5.0/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /logstash/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f ../conf/logstash.conf -v ``` And then strace: ``` [root@somwhere]# strace -p 19920 Process 19920 attached - interrupt to quit futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL Process 19920 detached [root@somwhere]# strace -p 19920 Process 19920 attached - interrupt to quit futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f098faebf00, FUTEX_WAKE_PRIVATE, 1) = 1 rt_sigreturn(0x7f098faebf00) = 202 futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f098faebf00, FUTEX_WAKE_PRIVATE, 1) = 1 rt_sigreturn(0x7f098faebf00) = 202 futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f098faebf00, FUTEX_WAKE_PRIVATE, 1) = 1 rt_sigreturn(0x7f098faebf00) = 202 futex(0x7f098eb509d0, FUTEX_WAIT, 19935, NULLPANIC: attached pid 19920 exited with 0 ``` It finally died. (I'm running it from root, because I'm going to add syslog input when its stable; from embedded systems, no port change option). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:39am UTC](https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944/7 "2017-07-06T05:39:29Z")