Logstash dont use *.conf files

attached
indent preformatted text by 4 spaces
root@logstashsapir:~# su - logstash
logstash@logstashsapir:~ /usr/share/logstash/bin/logstash "--path.settings" "/e tc/logstash" logstash@logstashsapir:~ /usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-05-30T12:12:22,983][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.7.0"}
[2020-05-30T12:12:27,499][INFO ][org.reflections.Reflections] Reflections took 93 ms to scan 1 urls, producing 21 keys and 41 values
[2020-05-30T12:12:31,663][ERROR][logstash.outputs.lumberjack] Invalid setting for lumberjack output plugin:

output {
lumberjack {
# This setting must be a path
# File does not exist or cannot be opened /usr/share/logstash/keys/TrustExternalCARoot.crt
ssl_certificate => "/usr/share/logstash/keys/TrustExternalCARoot.crt"
...
}
}
[2020-05-30T12:12:31,750][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:126)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:80)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:129
add at org/logstash/execution/ConvergeResultExt.java:57
converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:355
[2020-05-30T12:12:31,768][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in block in converge_state'"]} [2020-05-30T12:12:31,869][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateExceptionforPipelineAction::Create>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in block in converge_state'"]}
[2020-05-30T12:12:31,925][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
indent preformatted text by 4 spaces

using root account , give access to logstash user to that file.

chmod o+r /usr/share/logstash/keys/TrustExternalCARoot.crt

hi
i did give access to logstash
and tried loading logstash
both systemctl loading my config file and without config file using cli
indent preformatted text by 4 spaces
but no success

root@logstashsapir:~# chmod o+r /usr/share/logstash/keys/TrustExternalCARoot.crt
root@logstashsapir:~# systemctl stop logstash
root@logstashsapir:~# systemctl start logstash
root@logstashsapir:~# systemctl status logstash
? logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-05-30 15:00:55 UTC; 8s ago
Main PID: 8402 (java)
Tasks: 14 (limit: 2318)
CGroup: /system.slice/logstash.service
mq8402 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.hea

May 30 15:00:55 logstashsapir systemd[1]: Started logstash.
May 30 15:00:55 logstashsapir logstash[8402]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be re
May 30 15:00:59 logstashsapir logstash[8402]: WARNING: An illegal reflective access operation has occurred
May 30 15:00:59 logstashsapir logstash[8402]: WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-
May 30 15:00:59 logstashsapir logstash[8402]: WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
May 30 15:00:59 logstashsapir logstash[8402]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
May 30 15:00:59 logstashsapir logstash[8402]: WARNING: All illegal access operations will be denied in a future release
root@logstashsapir:~# netstat -taupn | grep ESTABLISHED
tcp 0 36 172.16.0.131:22 10.212.134.200:57291 ESTABLISHED 8272/sshd: logstash
root@logstashsapir:~#
root@logstashsapir:~#
root@logstashsapir:~#
root@logstashsapir:~#
root@logstashsapir:~# su - logstash
logstash@logstashsapir:~$ /usr/share/logstash/bin/logstash "--path.settings" "/e tc/logstash"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /e tc/logstash/log4j2.properties. Using default config which logs errors to the console
[FATAL] 2020-05-30 15:04:53.330 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:528:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:288:in validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:199:in block in validate_all'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:198:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:305:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:263:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in '"]}
[ERROR] 2020-05-30 15:04:53.354 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
indent preformatted text by 4 spaces

What is unclear about that?

ive tried that before the re-installation.
chmod logstash u+x /usr/share/logstash/keys

and this ive just tried
chmod o+r /usr/share/logstash/keys/TrustExternalCARoot.crt

do you mean of something else?

nowi see its a different dir
let me try that

ive just executed these two commands but still, and it starts working via cli (user logstash) but not via systemctl

indent preformatted text by 4 spaces 

sudo chmod 775 /usr/share/logstash/data/queue
sudo chmod 775 /usr/share/logstash/keys/

logstash@logstashsapir:~$ /usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-05-30T15:50:03,416][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.7.0"}
[2020-05-30T15:50:08,017][INFO ][org.reflections.Reflections] Reflections took 102 ms to scan 1 urls, producing 21 keys and 41 values
[2020-05-30T15:50:15,130][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-05-30T15:50:15,179][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/etc/logstash/conf.d/sflowlog.conf"], :thread=>"#<Thread:0x38a0d19b run>"}
[2020-05-30T15:50:16,907][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-05-30T15:50:17,176][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2020-05-30T15:50:17,219][INFO ][logstash.inputs.udp ][main][cfeabc89a9145c2e5083c84cd1cda4cd042fb5db41e7f4ceac3abee10d2deba8] Starting UDP listener {:address=>"0.0.0.0:6343"}
[2020-05-30T15:50:17,366][INFO ][logstash.inputs.udp ][main][cfeabc89a9145c2e5083c84cd1cda4cd042fb5db41e7f4ceac3abee10d2deba8] UDP listener started {:address=>"0.0.0.0:6343", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2020-05-30T15:50:17,734][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
indent preformatted text by 4 spaces

listening on right ports as configuration
indent preformatted text by 4 spaces
netstat -taupn | grep ESTABLISHED
tcp 0 0 172.16.0.131:22 10.212.134.200:57291 ESTABLISHED 8272/sshd: logstash
tcp 0 36 172.16.0.131:22 10.212.134.200:57814 ESTABLISHED 11012/sshd: logstas
tcp6 0 0 172.16.0.131:40210 23.22.183.192:5006 ESTABLISHED 11629/java
root@logstashsapir:~# netstat -unlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 0 0 127.0.0.53:53 0.0.0.0:* 966/systemd-resolve
udp 0 0 172.16.0.131:68 0.0.0.0:* 947/systemd-network
udp 0 0 0.0.0.0:6343 0.0.0.0:* 11629/java
indent preformatted text by 4 spaces

but not with systemctl
indent preformatted text by 4 spaces
systemctl start logstash
root@logstashsapir:~# netstat -taupn | grep ESTABLISHED
tcp 0 0 172.16.0.131:22 10.212.134.200:57291 ESTABLISHED 8272/sshd: logstash
tcp 0 36 172.16.0.131:22 10.212.134.200:57814 ESTABLISHED 11012/sshd: logstas
root@logstashsapir:~# netstat -taupn | grep ESTABLISHED
indent preformatted text by 4 spaces

running like this make it work just take a while.

indent preformatted text by 4 spaces

systemctl start logstash.service
indent preformatted text by 4 spaces

seems like make the dir writable did the work
thank you guys for all the help!!!!

do you know what went wrong with my installation?
because i did it by the guide, the only differences were the plugins: sflow and logz.io version of the lumberjack.

thank you again for all your trouble

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.