# I try to ingest the data from S3 bucket to opensearch dashboards using logstash **URL:** \ **Category:** Logstash\ **Tags:** elastic-stack-monitoring, elastic-stack-security\ **Created:** [June 4, 2024, 11:39am UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781 "2024-06-04T11:39:02Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 4, 2024, 11:39am UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/1 "2024-06-04T11:39:02Z")
[2024-06-04T11:09:21,409][INFO][logstash.javapipeline][main] Pipeline terminated {"pipeline.id"=\>"main"} [2024-06-04T11:09:21,484][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAction::Create, action\_result: false", :backtrace=\>nil} [2024-06-04T11:09:21,590][INFO][logstash.runner] Logstash shut down. [2024-06-04T11:09:21,624][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?] at opt.logstash\_minus\_8\_dot\_13\_dot\_4.lib.bootstrap.environment.(/opt/logstash-8.13.4/lib/bootstrap/environment.rb:90) ~[?:?] ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 4, 2024, 11:39am UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/2 "2024-06-04T11:39:02Z")
OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See [What is OpenSearch and the OpenSearch Dashboard? | Elastic](https://www.elastic.co/elasticsearch/opensearch) for more details. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 4, 2024, 12:01pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/3 "2024-06-04T12:01:15Z")
I suggest setting --log.level DEBUG to see if you get a more informative message. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 4, 2024, 12:08pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/4 "2024-06-04T12:08:19Z")
And also check journal: `journalctl -u logstash.service --since "20min ago"` ---
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 4, 2024, 1:44pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/5 "2024-06-04T13:44:28Z")
hi Badger, I install Opensearch , opensearch dashboard & logstash on EC2 server , i have a input log in s3 bucket . -bash-4.2# ./logstash -f config/pipelines.yml Using bundled JDK: /opt/logstash-8.13.4/jdk /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java\_thread\_pool\_executor.rb:13: warning: method redefined; discarding old to\_int /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java\_thread\_pool\_executor.rb:13: warning: method redefined; discarding old to\_f Sending Logstash logs to /opt/logstash-8.13.4/logs which is now configured via log4j2.properties [2024-06-04T13:43:29,713][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash-8.13.4/config/log4j2.properties [2024-06-04T13:43:29,715][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"8.13.4", "jruby.version"=\>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86\_64-linux]"} [2024-06-04T13:43:29,729][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11] [2024-06-04T13:43:29,731][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000` [2024-06-04T13:43:29,731][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000` [2024-06-04T13:43:31,319][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2024-06-04T13:43:35,877][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600, :ssl\_enabled=\>false} [2024-06-04T13:43:39,358][INFO][org.reflections.Reflections] Reflections took 1291 ms to scan 1 urls, producing 132 keys and 468 values /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/nokogiri-1.16.4-java/lib/nokogiri/xml/node.rb:1007: warning: method redefined; discarding old attr [2024-06-04T13:43:51,486][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=\>"output", :name=\>"opensearch"} [2024-06-04T13:43:51,537][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"Java::JavaLang::IllegalStateException", :message=\>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'opensearch'. Are you sure this is correct? Trying to load the opensearch output plugin resulted in this error: Unable to load the requested plugin named opensearch of type output. The plugin is not installed.", :backtrace=\>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1319)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET\_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET\_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET\_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:840)"]} [2024-06-04T13:43:51,699][INFO][logstash.runner] Logstash shut down. [2024-06-04T13:43:51,751][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?] at opt.logstash\_minus\_8\_dot\_13\_dot\_4.lib.bootstrap.environment.(/opt/logstash-8.13.4/lib/bootstrap/environment.rb:90) ~[?:?] ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 4, 2024, 2:04pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/6 "2024-06-04T14:04:53Z")
> [@Rathiga\_Thambu](#): > > ./logstash -f config/pipelines.yml Is that a YAML file? -f should point to a file (or directory of files) that define inputs, filters, and outputs. Not YAML. ---
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 4, 2024, 3:24pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/7 "2024-06-04T15:24:30Z")
I define inputs, filters, and outputs comment in pipelines.yml cd /opt/logstash.8.13.4/config vi pipelines.yml ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 4, 2024, 3:37pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/8 "2024-06-04T15:37:25Z")
Logstash configuration files are not yml files. The `pipelines.yml` is used to manage the pipelines that logstash will execute, while you can define the config string directly in the `pipelines.yml` you cannot use run logstash with the `-f` parameter pointing to this file. In fact running logstash with `-f` will make logstash ignore the `pipelines.yml`. If you want to run the pipelines defined in `pipelines.yml` you should run logstash as a service or using the command line `logstash --path.settings="/config/path"` where the `/config/path` is the path where `logstash.yml` and `pipelines.yml` are stored. ---
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 5, 2024, 4:25pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/9 "2024-06-05T16:25:17Z")
1. vi /opt/logstash-8.9.0/config/pipelines.yml [pipeline.id](http://pipeline.id/): main pipeline.workers: 1 pipeline.batch.size: 500 queue.type: persisted queue.max\_bytes: 1gb # Default = 1024mb queue.drain: true pipeline.output.workers: 1 path.config: "/opt/logstash-8.9.0/logstash-sample.conf" ---
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 5, 2024, 4:25pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/10 "2024-06-05T16:25:44Z")
cd /opt/logstash-8.9.0/config/logstash-sample.conf input { s3 { access\_key\_id =\> "AKIAUNWJFISALFXWOPVQ" secret\_access\_key =\> "1LeN8fzeq+AC1zs2SsyZtHsy+avPva5UgcVWvb8u" bucket =\> "serverlogs-01backup" region =\> "us-west-2" type =\> "s3" prefix =\> "appserver/rpmappserver-01/ribbideo\_logs/" interval =\> 60 } } output { opensearch { hosts =\> ["[http://35.91.61.103:9200](http://35.91.61.103:9200)"] index =\> "rpmappserver-01ribbideo\_logs" user =\> "admin" password =\> "admin" ssl\_certificate\_verification =\> false } } ---
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 5, 2024, 4:26pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/11 "2024-06-05T16:26:30Z")
-bash-4.2# ./logstash -f /config/pipelines.yml Using bundled JDK: /opt/logstash-8.9.0/jdk Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties [2024-06-05T16:03:20,112][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties [2024-06-05T16:03:20,114][INFO][logstash.runner] Starting Logstash {“logstash.version”=\>“8.9.0”, “jruby.version”=\>“jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86\_64-linux]”} [2024-06-05T16:03:20,148][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] [2024-06-05T16:03:21,754][WARN][logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified /opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated [2024-06-05T16:03:23,610][INFO][logstash.config.source.local.configpathloader] No config files found in path {:path=\>“/config/pipelines.yml”} [2024-06-05T16:03:23,640][ERROR][logstash.config.sourceloader] No configuration found in the configured sources. [2024-06-05T16:03:24,706][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600, :ssl\_enabled=\>false} [2024-06-05T16:03:24,741][INFO][logstash.runner] Logstash shut down. [2024-06-05T16:03:24,795][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?] at opt.logstash\_minus\_8\_dot\_9\_dot\_0.lib.bootstrap.environment.(/opt/logstash-8.9.0/lib/bootstrap/environment.rb:91) ~[?:?] ---
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 5, 2024, 4:27pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/12 "2024-06-05T16:27:32Z")
-bash-4.2# sudo systemctl status -l logstash.service ● logstash.service - Logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2024-06-05 16:19:32 UTC; 24s ago Docs: [Logstash Reference [8.14] | Elastic](https://www.elastic.co/guide/en/logstash/current/index.html) Main PID: 540 (java) CGroup: /system.slice/logstash.service └─540 /opt/logstash-8.9.0/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Djruby.regexp.interruptible=true -Djdk.io.File.enableADS=true --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -cp /opt/logstash-8.9.0/vendor/jruby/lib/jruby.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/checker-qual-3.12.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-codec-1.15.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-logging-1.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/error\_prone\_annotations-2.11.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/failureaccess-1.0.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/google-java-format-1.15.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/guava-31.1-jre.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/httpclient-4.5.13.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/httpcore-4.4.14.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/j2objc-annotations-1.3.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-annotations-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-core-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-databind-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-dataformat-yaml-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/janino-3.1.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jsr305-3.0.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jvm-options-parser-8.9.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-api-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-core-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/logstash-core.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/reflections-0.10.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/snakeyaml-1.33.jar org.logstash.Logstash -f /opt/logstash-8.9.0/config/pipelines.yml Jun 05 16:19:32 ip-172-31-4-236.us-west-2.compute.internal systemd[1]: Started Logstash. Jun 05 16:19:32 ip-172-31-4-236.us-west-2.compute.internal logstash[540]: Using bundled JDK: /opt/logstash-8.9.0/jdk -bash-4.2# ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 5, 2024, 4:34pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/13 "2024-06-05T16:34:29Z")
> [@Rathiga\_Thambu](#): > > ./logstash -f /config/pipelines.yml As mentioned, this will not work, `pipelines.yml` **is not** a configuration file. > [@Rathiga\_Thambu](#): > > sudo systemctl status -l logstash.service So, do you have any issue after running Logstash as a service? It is not clear what is the issue now. Also, please use the preformatted tex option, the `` button while sharing logs, it is pretty hard to read the unformatted text. ---
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 6, 2024, 5:18am UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/14 "2024-06-06T05:18:48Z")
I use logstash-sample.conf it shows same error -bash-4.2# ./logstash -f /config/logstash-sample.conf Using bundled JDK: /opt/logstash-8.9.0/jdk Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties [2024-06-06T05:03:16,629][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties [2024-06-06T05:03:16,648][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"8.9.0", "jruby.version"=\>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86\_64-linux]"} [2024-06-06T05:03:16,672][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] [2024-06-06T05:03:17,967][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified /opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated [2024-06-06T05:03:19,565][INFO][logstash.config.source.local.configpathloader] No config files found in path {:path=\>"/config/logstash-sample.conf"} [2024-06-06T05:03:19,566][ERROR][logstash.config.sourceloader] No configuration found in the configured sources. [2024-06-06T05:03:20,068][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600, :ssl\_enabled=\>false} [2024-06-06T05:03:20,135][INFO][logstash.runner] Logstash shut down. [2024-06-06T05:03:20,172][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?] at opt.logstash\_minus\_8\_dot\_9\_dot\_0.lib.bootstrap.environment.(/opt/logstash-8.9.0/lib/bootstrap/environment.rb:91) ~[?:?] ---
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 6, 2024, 5:19am UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/15 "2024-06-06T05:19:43Z")
pls help me to how to fix the error ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 6, 2024, 12:49pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/16 "2024-06-06T12:49:43Z")
You need to check the logs and troubleshoot it, they can give you some hints of the issue. Check these lines: > [2024-06-06T05:03:19,565][INFO][logstash.config.source.local.configpathloader] **No config files found** in path {:path=\>"/config/logstash-sample.conf"} > [2024-06-06T05:03:19,566][ERROR][logstash.config.sourceloader] **No configuration found in the configured sources**. This means that there is no configuration in the path you used, you probably used the wrong path, on another answer you shared this: > **/opt/logstash-8.9.0/config/logstash-sample.conf** So your `logstash-sample.conf` file is in the path `/opt/logstash-8.9.0/config/`, but you are telling logstash that is in the path `/config/logstash-sample.conf`, you need to pass the full correct path. ---
**Author:** ![Rathiga\_Thambu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rathiga_thambu/32/126785_2.png) [@Rathiga\_Thambu](https://discuss.elastic.co/u/Rathiga_Thambu)\ **Post date:** [June 6, 2024, 4:32pm UTC](https://discuss.elastic.co/t/i-try-to-ingest-the-data-from-s3-bucket-to-opensearch-dashboards-using-logstash/360781/17 "2024-06-06T16:32:05Z")
i change my file path : ````auto ```-bash-4.2# bin/logstash -f /opt/logstash-8.9.0/config/conf.d/pipe.conf Using bundled JDK: /opt/logstash-8.9.0/jdk Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties [2024-06-06T16:00:10,752][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties [2024-06-06T16:00:10,754][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.9.0", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]"} [2024-06-06T16:00:10,756][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] [2024-06-06T16:00:11,053][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified /opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated [2024-06-06T16:00:11,624][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [2024-06-06T16:00:12,313][INFO][org.reflections.Reflections] Reflections took 256 ms to scan 1 urls, producing 132 keys and 464 values [2024-06-06T16:00:14,986][INFO][logstash.javapipeline] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. [2024-06-06T16:00:15,006][INFO][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["http://35.91.61.103:9200"]} [2024-06-06T16:00:15,305][INFO][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://admin:xxxxxx@35.91.61.103:9200/]}} [2024-06-06T16:00:15,416][WARN][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://admin:xxxxxx@35.91.61.103:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://admin:xxxxxx@35.91.61.103:9200/][Manticore::ClientProtocolException] 35.91.61.103:9200 failed to respond"} [2024-06-06T16:00:15,450][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/opt/logstash-8.9.0/config/conf.d/pipe.conf"], :thread=>"#"} [2024-06-06T16:00:16,490][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=>1.04} [2024-06-06T16:00:16,518][INFO][logstash.inputs.s3][main] Registering {:bucket=>"serverlogs-01backup", :region=>"us-west-2"} [2024-06-06T16:00:16,679][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} [2024-06-06T16:00:16,689][INFO][logstash.inputs.s3][main][9d90c0eabc646bbe45308fb6e092089a776e0d44b6774524f913d1aa89dab68f] Using default generated file for the sincedb {:filename=>"/opt/logstash-8.9.0/data/plugins/inputs/s3/sincedb_86f0e9b9f3fbe965c79680c4ad84fb16"} [2024-06-06T16:00:16,706][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2024-06-06T16:00:20,594][WARN][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://admin:xxxxxx@35.91.61.103:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://admin:xxxxxx@35.91.61.103:9200/][Manticore::ClientProtocolException] 35.91.61.103:9200 failed to respond"} ````