I try to ingest the data from S3 bucket to opensearch dashboards using logstash

Elastic Stack Logstash
,
1

[2024-06-04T11:09:21,409][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2024-06-04T11:09:21,484][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2024-06-04T11:09:21,590][INFO ][logstash.runner ] Logstash shut down.
[2024-06-04T11:09:21,624][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
at opt.logstash_minus_8_dot_13_dot_4.lib.bootstrap.environment.(/opt/logstash-8.13.4/lib/bootstrap/environment.rb:90) ~[?:?]

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See What is OpenSearch and the OpenSearch Dashboard? | Elastic for more details.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

I suggest setting --log.level DEBUG to see if you get a more informative message.

1 Like
4

And also check journal:
journalctl -u logstash.service --since "20min ago"

5

hi Badger,
I install Opensearch , opensearch dashboard & logstash on EC2 server , i have a input log in s3 bucket .
-bash-4.2# ./logstash -f config/pipelines.yml
Using bundled JDK: /opt/logstash-8.13.4/jdk
/opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Sending Logstash logs to /opt/logstash-8.13.4/logs which is now configured via log4j2.properties
[2024-06-04T13:43:29,713][INFO ][logstash.runner ] Log4j configuration path used is: /opt/logstash-8.13.4/config/log4j2.properties
[2024-06-04T13:43:29,715][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.13.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"}
[2024-06-04T13:43:29,729][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-06-04T13:43:29,731][INFO ][logstash.runner ] Jackson default value override logstash.jackson.stream-read-constraints.max-string-length configured to 200000000
[2024-06-04T13:43:29,731][INFO ][logstash.runner ] Jackson default value override logstash.jackson.stream-read-constraints.max-number-length configured to 10000
[2024-06-04T13:43:31,319][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2024-06-04T13:43:35,877][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-06-04T13:43:39,358][INFO ][org.reflections.Reflections] Reflections took 1291 ms to scan 1 urls, producing 132 keys and 468 values
/opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/nokogiri-1.16.4-java/lib/nokogiri/xml/node.rb:1007: warning: method redefined; discarding old attr
[2024-06-04T13:43:51,486][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"output", :name=>"opensearch"}
[2024-06-04T13:43:51,537][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'opensearch'. Are you sure this is correct? Trying to load the opensearch output plugin resulted in this error: Unable to load the requested plugin named opensearch of type output. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1319)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
[2024-06-04T13:43:51,699][INFO ][logstash.runner ] Logstash shut down.
[2024-06-04T13:43:51,751][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
at opt.logstash_minus_8_dot_13_dot_4.lib.bootstrap.environment.(/opt/logstash-8.13.4/lib/bootstrap/environment.rb:90) ~[?:?]

6

Is that a YAML file? -f should point to a file (or directory of files) that define inputs, filters, and outputs. Not YAML.

7

I define inputs, filters, and outputs comment in pipelines.yml
cd /opt/logstash.8.13.4/config
vi pipelines.yml

8

Logstash configuration files are not yml files.

The pipelines.yml is used to manage the pipelines that logstash will execute, while you can define the config string directly in the pipelines.yml you cannot use run logstash with the -f parameter pointing to this file.

In fact running logstash with -f will make logstash ignore the pipelines.yml.

If you want to run the pipelines defined in pipelines.yml you should run logstash as a service or using the command line logstash --path.settings="/config/path" where the /config/path is the path where logstash.yml and pipelines.yml are stored.

9
  1. vi /opt/logstash-8.9.0/config/pipelines.yml

pipeline.id: main
pipeline.workers: 1
pipeline.batch.size: 500
queue.type: persisted
queue.max_bytes: 1gb # Default = 1024mb
queue.drain: true
pipeline.output.workers: 1
path.config: "/opt/logstash-8.9.0/logstash-sample.conf"

10

cd /opt/logstash-8.9.0/config/logstash-sample.conf

input {
s3 {
access_key_id => "AKIAUNWJFISALFXWOPVQ"
secret_access_key => "1LeN8fzeq+AC1zs2SsyZtHsy+avPva5UgcVWvb8u"
bucket => "serverlogs-01backup"
region => "us-west-2"
type => "s3"
prefix => "appserver/rpmappserver-01/ribbideo_logs/"
interval => 60
}
}

output {
opensearch {
hosts => ["http://35.91.61.103:9200"]
index => "rpmappserver-01ribbideo_logs"
user => "admin"
password => "admin"
ssl_certificate_verification => false
}
}

11

-bash-4.2# ./logstash -f /config/pipelines.yml
Using bundled JDK: /opt/logstash-8.9.0/jdk
Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties
[2024-06-05T16:03:20,112][INFO ][logstash.runner ] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties
[2024-06-05T16:03:20,114][INFO ][logstash.runner ] Starting Logstash {“logstash.version”=>“8.9.0”, “jruby.version”=>“jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]”}
[2024-06-05T16:03:20,148][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-06-05T16:03:21,754][WARN ][logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified
/opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated
[2024-06-05T16:03:23,610][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>“/config/pipelines.yml”}
[2024-06-05T16:03:23,640][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2024-06-05T16:03:24,706][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-06-05T16:03:24,741][INFO ][logstash.runner ] Logstash shut down.
[2024-06-05T16:03:24,795][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at opt.logstash_minus_8_dot_9_dot_0.lib.bootstrap.environment.(/opt/logstash-8.9.0/lib/bootstrap/environment.rb:91) ~[?:?]

12

-bash-4.2# sudo systemctl status -l logstash.service
● logstash.service - Logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2024-06-05 16:19:32 UTC; 24s ago
Docs: Logstash Reference [8.14] | Elastic
Main PID: 540 (java)
CGroup: /system.slice/logstash.service
└─540 /opt/logstash-8.9.0/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Djruby.regexp.interruptible=true -Djdk.io.File.enableADS=true --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -cp /opt/logstash-8.9.0/vendor/jruby/lib/jruby.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/checker-qual-3.12.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-codec-1.15.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-logging-1.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/error_prone_annotations-2.11.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/failureaccess-1.0.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/google-java-format-1.15.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/guava-31.1-jre.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/httpclient-4.5.13.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/httpcore-4.4.14.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/j2objc-annotations-1.3.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-annotations-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-core-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-databind-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-dataformat-yaml-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/janino-3.1.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jsr305-3.0.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jvm-options-parser-8.9.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-api-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-core-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/logstash-core.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/reflections-0.10.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/snakeyaml-1.33.jar org.logstash.Logstash -f /opt/logstash-8.9.0/config/pipelines.yml

Jun 05 16:19:32 ip-172-31-4-236.us-west-2.compute.internal systemd[1]: Started Logstash.
Jun 05 16:19:32 ip-172-31-4-236.us-west-2.compute.internal logstash[540]: Using bundled JDK: /opt/logstash-8.9.0/jdk
-bash-4.2#

13

As mentioned, this will not work, pipelines.yml is not a configuration file.

So, do you have any issue after running Logstash as a service?

It is not clear what is the issue now.

Also, please use the preformatted tex option, the </> button while sharing logs, it is pretty hard to read the unformatted text.

14

I use logstash-sample.conf it shows same error

-bash-4.2# ./logstash -f /config/logstash-sample.conf
Using bundled JDK: /opt/logstash-8.9.0/jdk
Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties
[2024-06-06T05:03:16,629][INFO ][logstash.runner ] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties
[2024-06-06T05:03:16,648][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.9.0", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]"}
[2024-06-06T05:03:16,672][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-06-06T05:03:17,967][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
/opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated
[2024-06-06T05:03:19,565][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/config/logstash-sample.conf"}
[2024-06-06T05:03:19,566][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2024-06-06T05:03:20,068][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-06-06T05:03:20,135][INFO ][logstash.runner ] Logstash shut down.
[2024-06-06T05:03:20,172][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at opt.logstash_minus_8_dot_9_dot_0.lib.bootstrap.environment.(/opt/logstash-8.9.0/lib/bootstrap/environment.rb:91) ~[?:?]

15

pls help me to how to fix the error

16

You need to check the logs and troubleshoot it, they can give you some hints of the issue.

Check these lines:

[2024-06-06T05:03:19,565][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/config/logstash-sample.conf"}
[2024-06-06T05:03:19,566][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.

This means that there is no configuration in the path you used, you probably used the wrong path, on another answer you shared this:

/opt/logstash-8.9.0/config/logstash-sample.conf

So your logstash-sample.conf file is in the path /opt/logstash-8.9.0/config/, but you are telling logstash that is in the path /config/logstash-sample.conf, you need to pass the full correct path.

17

i change my file path :

```-bash-4.2# bin/logstash -f /opt/logstash-8.9.0/config/conf.d/pipe.conf
Using bundled JDK: /opt/logstash-8.9.0/jdk
Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties
[2024-06-06T16:00:10,752][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties
[2024-06-06T16:00:10,754][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.9.0", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]"}
[2024-06-06T16:00:10,756][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-06-06T16:00:11,053][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
/opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated
[2024-06-06T16:00:11,624][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-06-06T16:00:12,313][INFO ][org.reflections.Reflections] Reflections took 256 ms to scan 1 urls, producing 132 keys and 464 values
[2024-06-06T16:00:14,986][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2024-06-06T16:00:15,006][INFO ][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["http://35.91.61.103:9200"]}
[2024-06-06T16:00:15,305][INFO ][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://admin:xxxxxx@35.91.61.103:9200/]}}
[2024-06-06T16:00:15,416][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://admin:xxxxxx@35.91.61.103:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://admin:xxxxxx@35.91.61.103:9200/][Manticore::ClientProtocolException] 35.91.61.103:9200 failed to respond"}
[2024-06-06T16:00:15,450][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/opt/logstash-8.9.0/config/conf.d/pipe.conf"], :thread=>"#<Thread:0x59e80928@/opt/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-06-06T16:00:16,490][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.04}
[2024-06-06T16:00:16,518][INFO ][logstash.inputs.s3       ][main] Registering {:bucket=>"serverlogs-01backup", :region=>"us-west-2"}
[2024-06-06T16:00:16,679][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2024-06-06T16:00:16,689][INFO ][logstash.inputs.s3       ][main][9d90c0eabc646bbe45308fb6e092089a776e0d44b6774524f913d1aa89dab68f] Using default generated file for the sincedb {:filename=>"/opt/logstash-8.9.0/data/plugins/inputs/s3/sincedb_86f0e9b9f3fbe965c79680c4ad84fb16"}
[2024-06-06T16:00:16,706][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2024-06-06T16:00:20,594][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://admin:xxxxxx@35.91.61.103:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://admin:xxxxxx@35.91.61.103:9200/][Manticore::ClientProtocolException] 35.91.61.103:9200 failed to respond"}