# Multiple pipelines utilizing s3 output cause each other to terminate **URL:** \ **Category:** Logstash\ **Created:** [June 16, 2023, 10:39am UTC](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211 "2023-06-16T10:39:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![hughjarse](https://avatars.discourse-cdn.com/v4/letter/h/47e85d/32.png) [@hughjarse](https://discuss.elastic.co/u/hughjarse)\ **Post date:** [June 16, 2023, 10:39am UTC](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211/1 "2023-06-16T10:39:56Z")
In Logstash, I am using the s3 output plugin in multiple pipelines. Each plugin instance stores temporary files to disk and has the `restore` option set to `true`. I have problems with the restarting of **other** pipelines causing the non-restarting pipeline's temporary files to be deleted and then the pipeline terminates when the restore feature cannot find the deleted temporary file. Disabling `restore` for the s3 outputs of course avoids this problem. Additionally, when all of the pipelines' configurations are squeezed into a single pipeline, the problem does not occur (of course, there's only one pipeline that's restarting 😉 ) **Is this multi-pipeline use case a known problem for the s3 output plugin? Otherwise, how can I avoid this problem from happening?** For example, Pipelines _ABC_ and _XYZ_ both use the s3 output plugin. When _Pipeline ABC_ restarts, I see the temporary file for _Pipeline XYZ_ get deleted even though the temporary file's 5-min rotation time has not yet been reached. Logstash's log then states that as part of _Pipeline ABC's_ restart, the "corrupted" temporary file for _Pipeline XYZ_ has been "recovered" (in reality, it no longer exists). ```auto (snipped) Jun 16 18:35:38 logstashHost logstash: [2023-06-16T18:35:38,927][INFO][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:"ABC"} Jun 16 18:35:59 logstashHost logstash: [2023-06-16T18:35:59,877][WARN][org.logstash.plugins.integration.outputs.s3.GzipUtil][ABC] Corrupted file recovered, path:/var/lib/logstash/s3-output-tmp/6b9aec2a-5f52-4822-b7cb-5f31a22a2d33/PIPELINE-XYZ-LOGS/2023/06/16/ls.s3.8bdd9841-2c1c-4042-bb66-1d624e838e35.2023-06-16T18.35.part1-recovered.txt Jun 16 18:36:20 logstashHost logstash: [2023-06-16T18:36:20,860][INFO][logstash.javapipeline][ABC] Pipeline started {"pipeline.id"=>"ABC"} Jun 16 18:36:20 logstashHost logstash: [2023-06-16T18:36:20,865][INFO][logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:XYZ, :"ABC"], :non_running_pipelines=>[]} ``` A few minutes later, _Pipeline XYZ_ now terminates because the recovery of its s3 output temporary file fails due to its non-existence. ```auto (snipped) Jun 16 18:40:59 logstashHost logstash: [2023-06-16T18:40:59,641][ERROR][logstash.javapipeline][XYZ] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"XYZ", :error=>"(IOError) finished", :exception=>Java::OrgJrubyExceptions::IOError, :backtrace=>["org.jruby.ext.zlib.JZlibRubyGzipWriter.write(org/jruby/ext/zlib/JZlibRubyGzipWriter.java:376)", "usr.share.logstash.vendor.jruby.lib.ruby.stdlib.forwardable.write(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/forwardable.rb:230)", "usr.share.logstash.vendor.jruby.lib.ruby.stdlib.forwardable.write(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/forwardable.rb:230)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_1_minus_java.lib.logstash.outputs.s3.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3.rb:248)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_1_minus_java.lib.logstash.outputs.s3.file_repository.get_file(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_1_minus_java.lib.logstash.outputs.s3.file_repository.get_factory(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:107)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_1_minus_java.lib.logstash.outputs.s3.file_repository.with_lock(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:27)", "org.jruby.ext.monitor.Monitor.synchronize(org/jruby/ext/monitor/Monitor.java:82)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_1_minus_java.lib.logstash.outputs.s3.file_repository.with_lock(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:26)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_1_minus_java.lib.logstash.outputs.s3.file_repository.get_factory(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:103)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_1_minus_java.lib.logstash.outputs.s3.file_repository.get_file(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_1_minus_java.lib.logstash.outputs.s3.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3.rb:248)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_1_minus_java.lib.logstash.outputs.s3.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3.rb:243)", "usr.share.logstash.logstash_minus_core.lib.logstash.outputs.base.multi_receive(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:103)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304)"], :thread=>"#"} Jun 16 18:41:02 logstashHost logstash: [2023-06-16T18:41:02,951][ERROR][logstash.javapipeline][XYZ] Pipeline error {:pipeline_id=>"XYZ", :exception=>#, :backtrace=>["org/jruby/RubyFileTest.java:248:in `size'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/temporary_file.rb:50:in `size'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3.rb:369:in `upload_file'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3.rb:274:in `block in close'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:89:in `block in each_files'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:132:in `block in each_factory'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:27:in `block in with_lock'", "org/jruby/ext/monitor/Monitor.java:82:in `synchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:26:in `with_lock'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:131:in `block in each_factory'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:129:in `each_factory'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3/file_repository.rb:88:in `each_files'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.1-java/lib/logstash/outputs/s3.rb:273:in `close'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:98:in `do_close'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:75:in `do_close'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:483:in `shutdown_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:209:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/pipeline_XYZ/XYZ.conf"], :thread=>"#"} Jun 16 18:41:02 logstashHost logstash: [2023-06-16T18:41:02,952][INFO][logstash.javapipeline][XYZ] Pipeline terminated {"pipeline.id"=>"XYZ"} ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 16, 2023, 1:14pm UTC](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211/2 "2023-06-16T13:14:29Z")
Can you share your pipeline configurations? Also, in the logs you shared the temporary directory is the same for both pipelines, did this issue still happens if you use a different tmp directory for each pipeline? ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 17, 2023, 10:57am UTC](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211/3 "2023-06-17T10:57:08Z")
Which version of LS are you running? Check [this](https://github.com/logstash-plugins/logstash-output-s3/issues/187). ---
**Author:** ![hughjarse](https://avatars.discourse-cdn.com/v4/letter/h/47e85d/32.png) [@hughjarse](https://discuss.elastic.co/u/hughjarse)\ **Post date:** [June 21, 2023, 12:12pm UTC](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211/4 "2023-06-21T12:12:22Z")
So far from my testing, @leandrojmp's suggestion to use a different tmp directory for each pipeline (specifically, a different tmp directory for each s3 output _block_ within each pipeline -- some pipelines have configs that include multiple s3 blocks based on conditionals; a leftover from when the configs were all part of a single pipeline) seems to have resolved the issue. The symptoms were nearly identical to those in the GitHub ticket that @Rios shared (as well as [this other one](https://github.com/logstash-plugins/logstash-output-s3/issues/107)), which is interesting since there are references in those tickets that this problem was supposedly fixed in earlier versions of the output plugins. For posterity's sake, I was using the following versions: - Logstash 8.8.1 (rpm-based package) - logstash-output-s3 as bundled within logstash-integration-aws 7.1.1 My problematic configs were like the following... Pipeline ABC: ```auto output { if [type] == "ABC-logs" { s3 { region => "ap-northeast-1" bucket => "MyBucketName" prefix => "ABC-logs/%{+YYYY}/%{+MM}/%{+dd}" temporary_directory => "/var/lib/logstash/s3-output-tmp" encoding => "gzip" rotation_strategy => "time" time_file => 5 restore => true } } #...one other lumberjack output snipped... } ``` Pipeline XYZ: ```auto output { if [type] == "FOO-logs" { s3 { region => "ap-northeast-1" bucket => "MyBucketName" prefix => "FOO-logs/%{+YYYY}/%{+MM}/%{+dd}" temporary_directory => "/var/lib/logstash/s3-output-tmp" encoding => "gzip" rotation_strategy => "time" time_file => 5 restore => true } } if [type] == "BAR-logs" { s3 { region => "ap-northeast-1" bucket => "MyBucketName" prefix => "BAR-logs/%{+YYYY}/%{+MM}/%{+dd}" temporary_directory => "/var/lib/logstash/s3-output-tmp" encoding => "gzip" rotation_strategy => "time" time_file => 5 restore => true } } #...one other lumberjack output snipped... } ``` As mentioned above, configuring each s3 block to use a **unique** `temporary_directory` seems to avoid the problem. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 21, 2023, 6:44pm UTC](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211/6 "2023-06-21T18:44:22Z")
The code is blowing up [here](https://github.com/logstash-plugins/logstash-integration-aws/blob/8532ad0060748f38c8eb92aa90efbe4443933bf9/lib/logstash/outputs/s3.rb#L274). The outputs are shutting down, and each one is iterating over the files in the temporary directory to upload them to s3. I find it entirely unsurprising that there is a race condition where two outputs try to delete the same file and one of them fails. Plus it is a documented requirement for each output to have its own temporary directory. See the second "Important!" in [this](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html#_description_102) section of the documentation. ---
**Author:** ![hughjarse](https://avatars.discourse-cdn.com/v4/letter/h/47e85d/32.png) [@hughjarse](https://discuss.elastic.co/u/hughjarse)\ **Post date:** [June 22, 2023, 12:54am UTC](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211/7 "2023-06-22T00:54:51Z")
**Many** thanks for pointing both of these out! > Plus it is a documented requirement for each output to have its own temporary directory. See the second "Important!" in [this](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html#_description_102) section of the documentation. \ **Re-Note to Self (nth time):** when troubleshooting, Reading The Fine Manual _again_ is more efficient -- and enlightening -- than bashing your head against the wall... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 20, 2023, 12:55am UTC](https://discuss.elastic.co/t/multiple-pipelines-utilizing-s3-output-cause-each-other-to-terminate/336211/8 "2023-07-20T00:55:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.