Logstash pipeline is not ingesting data into the elasticsearch index though the pipeline was running. This pipeline was deployed one
year back and it was running well since then. But on 24th May 2021, it stopped ingesting data. Restarting the logstash fixed the issue.
we have checked logstash logs but did find nothing there. Please see the below log.
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,503][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>http://localhost:9200/}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,619][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,623][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
Aug 26 12:24:22 xyz.com systemd[1]: Stopping logstash...
Aug 26 12:24:22 xyz.com logstash[827]: [2020-08-26T12:24:22,665][WARN ][logstash.runner ] SIGTERM received. Shutting down.
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,864][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>33, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.9-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>25, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>26, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>27, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>28, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>29, "name"=>"[main]>worker4", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>30, "name"=>"[main]>worker5", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>31, "name"=>"[main]>worker6", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>32, "name"=>"[main]>worker7", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}]}}
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,866][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,852][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x2c879c55 run>"}
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,855][INFO ][logstash.runner ] Logstash shut down.
Aug 26 12:24:29 xyz.com systemd[1]: Stopped logstash.
May 24 08:11:46 xyz.com systemd[1]: Started logstash.
May 24 08:12:11 xyz.com logstash[19174]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 08:12:11 xyz.com logstash[19174]: 2021-05-24 08:12:11,794 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.open0(Native Method)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
May 24 08:12:11 xyz.com logstash[19174]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:640)
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,120][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,363][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Main process exited, code=killed, status=9/KILL
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Failed with result 'signal'.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
May 24 15:38:50 xyz.com systemd[1]: Stopped logstash.
May 24 15:38:50 xyz.com systemd[1]: Started logstash.
May 24 15:39:06 xyz.com logstash[25666]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 15:39:07 xyz.com logstash[25666]: 2021-05-24 15:39:07,101 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFo
May 24 15:39:07 xyz.com logstash[25666]: at java.io.FileOutputStream.open0(Native Method)
May 24 15:39:07 xyz.com logstash[25666]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
Server OS: Ubuntu 18.04 ELK Version: 7.11.1
Need your help to find out the exact reason.