Logstash dont use *.conf files

hi all

i have an issue using configuration files that are under /etc/logstash/conf.d directory.
The issue is that when loading logstash using systemctl the demon don't pull the configuration from the *.conf file under /etc/logstash/conf.d/
only when loading via cli this configuration pulls.
ive verified that my pipeline.yml is pointing to this config directory
/

  • pipeline.id: main
    path.config: "/etc/logstash/conf.d/*.conf"
    /

and that my logstash.yml the main pipeline is enabled
/
pipeline.id: main
/

permission wise all files owned by root
but i installed my logstash as root (sudo bash)
ive tried to execute "chmod 644 *" in my directories but it didn't help

any idea how to solve it?
thanks

which user does the logstash daemon run as ? that user needs access to your conf files.

when running from cli, which user do you use ?

hi

the logstash daemon run as logstash.
via cli i run it as root.
ive try changing the permission for conf.d folder
chmod -R a+rwx /etc/logstash/conf.d/
that didnt help

could be this issue is because ive installed logstash after executing "sudo bash"?

/ ls -la
total 56
drwxrwxr-x 3 root root 4096 May 26 08:09 .
drwxr-xr-x 95 root root 4096 May 26 06:36 ..
drwxrwxrwx 2 root root 4096 May 26 08:07 conf.d/

//usr/share/logstash# ls -la
total 684
drwxrwxr-x 12 logstash logstash 4096 May 20 12:07 .
drwxr-xr-x 117 root root 4096 May 20 11:54 ..
drwxrwxr-x 2 logstash logstash 4096 May 20 11:54 bin
-rw-r--r-- 1 logstash logstash 2276 May 12 04:31 CONTRIBUTORS
drwxrwxr-x 4 logstash logstash 4096 May 20 14:32 data
-rw-r--r-- 1 logstash logstash 4068 May 21 10:49 Gemfile
-rw-r--r-- 1 logstash logstash 22899 May 21 10:50 Gemfile.lock
drwxr-x--- 2 root root 4096 May 20 11:54 keys
/

try switching to logstash user and run from CLI to see where the error is. you might have to temporaril give shell access and home directory to logstash user.

hi
ive tried to run it as logstash user, but got "This account is currently not available"
/
su logstash -c '/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sflowlog.confls'
/

so ive tried with a differnt usr (how is not root but in sudo group (logstashadmin)

and this is the output /
su logstash -c '/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sflowlog.confls'
This account is currently not available.
root@logstash-sapir:~# su logstashadmin -c '/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sflowlog.confls'
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[FATAL] 2020-05-28 03:17:52.506 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:528:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:288:in validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:199:in block in validate_all'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:198:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:305:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:263:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in '"]}
[ERROR] 2020-05-28 03:17:52.540 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit/

ive tried to change in "vi /etc/systemd/system/logstash.service"
and execut "systemctl daemon-reload"
the user and group to root, but still the service dont use the config file (or dont really come up)

ive changed "/usr/share/logstash/data" to be writable "chmod -R a+rwX /usr/share/logstash/data".

but ive noticed i had a typo when loading cli logstash so i execute it again
and i got a long log.
/su logstashadmin -c '/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sflowlog.conf'
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-05-28 03:33:00.788 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-05-28 03:33:00.823 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.7.0"}
[INFO ] 2020-05-28 03:33:09.557 [Converge PipelineAction::Create] Reflections - Reflections took 203 ms to scan 1 urls, producing 21 keys and 41 values
[ERROR] 2020-05-28 03:33:17.604 [Converge PipelineAction::Create] lumberjack - Invalid setting for lumberjack output plugin:

output {
lumberjack {
# This setting must be a path
# File does not exist or cannot be opened /usr/share/logstash/keys/TrustExternalCARoot.crt
ssl_certificate => "/usr/share/logstash/keys/TrustExternalCARoot.crt"
...
}
}
[ERROR] 2020-05-28 03:33:17.757 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:126)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:80)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:129
add at org/logstash/execution/ConvergeResultExt.java:57
converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:355
[ERROR] 2020-05-28 03:33:17.779 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in block in converge_state'"]} [FATAL] 2020-05-28 03:33:17.872 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateExceptionforPipelineAction::Create>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in block in converge_state'"]}
[ERROR] 2020-05-28 03:33:17.952 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit/

basically it saying two things:

  1. that something is wrong with my lumberjack settings, but it working as root.
    lumberjack {
    hosts => ["listener.logz.io"]
    port => 5006
    ssl_certificate => "/usr/share/logstash/keys/TrustExternalCARoot.crt"
    codec => "json_lines"
    }

  2. and that something is wrong with my java

/runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in `block in converge_state'"]}/

and again thank you for all the help im really new to this and straggling.
it feels like my installation has done wrongly.
ive preform the installation as root, maybe this is the issue?

by default logstash user don’t have home directory and shell. you will need to give the user home and shell. for example (run this as root)

#usermod -d /usr/share/logstash logstash
#usermod -s /bin/bash logstash

you shouldn’t need to modify permissions as package manager handles that portion.

can you show the content of logstash.service in /etc-systemd , the permission of /etc/logstash, and the output of cat /etc/passwd | grep logstash

also, are you using package management like yum or apt?

hi
im using apt

how ive installed logstash

/ apt update
java -version
apt install default-jre
java -version
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt-get update && sudo apt-get install logstash
/

vi /etc/systemd/system/logstash.service

/[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash

Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.

Prefixing the path with '-' makes it try to load, but if the file doesn't

exist, it continues onward.

EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

When stopping, how long to wait before giving up and sending SIGKILL?

Keep in mind that SIGKILL on a process can cause data loss.

TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target/

permission of /etc/logstash
/
root@logstash-sapir:~# ls -la /etc/logstash
total 56
drwxrwxr-x 3 root root 4096 May 26 08:09 .
drwxr-xr-x 95 root root 4096 May 28 03:10 ..
drwxrwxrwx 2 root root 4096 May 28 03:45 conf.d
-rw-r--r-- 1 root root 2019 May 12 04:31 jvm.options
-rw-r--r-- 1 root root 8958 May 12 04:31 log4j2.properties
-rw-r--r-- 1 root root 342 May 12 04:31 logstash-sample.conf
-rw-r--r-- 1 root root 9475 May 26 06:24 logstash.yml
-rw-r--r-- 1 root root 285 May 12 04:31 pipelines.yml
-rw-r--r-- 1 root root 753 May 24 13:34 sflowlog.yml
-rw-r--r-- 1 root root 1696 May 12 04:31 startup.options
/

oot@logstash-sapir:~# cat /etc/passwd | grep logstash
/
logstashadmin:x:1000:1000:logstash:/home/logstashadmin:/bin/bash
logstash:x:999:999:LogStash Service User:/usr/share/logstash:/usr/sbin/nologin
/

next time please use markdown for providing files for easier reading. you can wrap config using the </> button or using ``` before and after the config.
going back to your problem :

  1. give logstash shell access by
    usermod -s /bin/bash logstash

  2. switch user to logstash (su - logstash)

  3. run logstash from CLI using logstash user and provide the error message that you encounter. don't give any config parameters, just do :

/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"

hi
back to my computer.
ive executed as you suggested.

    logstashadmin@logstash-sapir:~$ su - logstash
Password:
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
    indent preformatted text by 4 spaces
logstash@logstash-sapir:~$ /usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
ERROR: Failed to parse YAML file "/etc/logstash/logstash.yml". Please confirm if the YAML structure is valid (e.g. look for incorrect usage of whitespace or indentation). Aborting... parser_error=>(<unknown>): expected <block end>, but found '<block mapping start>' while parsing a block mapping at line 34 column 2
[ERROR] 2020-05-28 13:14:35.352 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit`Preformatted text`

Preformatted text

`

there is an error in your logstash.yml. can you show the content or check that it's formatted correctly?

the line ' pipeline.id: main' was uncomment so i comment it back.
now all is commented but this line 'path.logs: /var/log/logstash'

when im running the command now i get

'/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-05-28T15:47:08,558][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.7.0"}
[2020-05-28T15:47:20,115][INFO ][org.reflections.Reflections] Reflections took 184 ms to scan 1 urls, producing 21 keys and 41 values
[2020-05-28T15:47:28,981][ERROR][logstash.outputs.lumberjack] Invalid setting for lumberjack output plugin:

output {
lumberjack {
# This setting must be a path
# File does not exist or cannot be opened /usr/share/logstash/keys/TrustExternalCARoot.crt
ssl_certificate => "/usr/share/logstash/keys/TrustExternalCARoot.crt"
...
}
}
[2020-05-28T15:47:29,151][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:126)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:80)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:129
add at org/logstash/execution/ConvergeResultExt.java:57
converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:355
[2020-05-28T15:47:29,191][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in block in converge_state'"]} [2020-05-28T15:47:29,282][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateExceptionforPipelineAction::Create>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in block in converge_state'"]}
[2020-05-28T15:47:29,396][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
'

Does the logstash user have access to that directory and that file?

ive reinstall it from screech gave all access to the key directory
Preformatted textsudo chmod -R ugo+rw /usr/share/logstash/keys/

indent preformatted text by 4 spaces

gave logstash user shell access

but still when i run as root all work even my conf file
but when i run in other users (in this case logstash admin due to the fact logstash user dont have pass it created by apt-get installation.
ive got this really long error
ive run it with
indent preformatted text by 4 spaces
./logstash "--path.settings" "/etc/logstash"
indent preformatted text by 4 spaces
indent preformatted text by 4 spaces

logstashadmin@logstashsapir:/usr/share/logstash/bin$ ./usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
-bash: ./usr/share/logstash/bin/logstash: No such file or directory
logstashadmin@logstashsapir:/usr/share/logstash/bin$ ./logstash "--path.settings" "/etc/logstash"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
2020-05-30 05:00:18,955 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:237)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:158)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:664)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:631)
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:205)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:146)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:62)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:64)
at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:36)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:272)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:408)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:293)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:647)
at org.apache.logging.log4j.core.LoggerContext.setConfigLocation(LoggerContext.java:637)
at org.logstash.log.LoggerExt.reconfigure(LoggerExt.java:180)
at org.logstash.log.LoggerExt$INVOKER$s$1$0$reconfigure.call(LoggerExt$INVOKER$s$1$0$reconfigure.gen)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:375)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:174)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_0_dot_6_dot_5.lib.clamp.command.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_0_dot_6_dot_5.lib.clamp.command.RUBY$method$run$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)
at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)
at org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)
at usr.share.logstash.logstash_minus_core.lib.logstash.runner.RUBY$method$run$0(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:263)
at usr.share.logstash.logstash_minus_core.lib.logstash.runner.RUBY$method$run$0$VARARGS(/usr/share/logstash/logstash-core/lib/logstash/runner.rb)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_0_dot_6_dot_5.lib.clamp.command.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
at usr.share.logstash.lib.bootstrap.environment.RUBY$script(/usr/share/logstash/lib/bootstrap/environment.rb:88)
at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
at org.jruby.ir.Compiler$1.load(Compiler.java:89)
at org.jruby.Ruby.runScript(Ruby.java:1205)
at org.jruby.Ruby.runNormally(Ruby.java:1128)
at org.jruby.Ruby.runNormally(Ruby.java:1146)
at org.jruby.Ruby.runFromMain(Ruby.java:958)
at org.logstash.Logstash.run(Logstash.java:133)
at org.logstash.Logstash.main(Logstash.java:67)
indent preformatted text by 4 spaces

it was really long so ive passed only part of it

this error here shows that the user you run logstash with doesn’t have permission to the log files in /var/log/logstash.

logstash user does not have a password by default because it is not supposed to be used.

now that you have reinstall logstash, you can try redoing this. also please don’t change permissions manually, it tends to mess up things

it asking for pass
indent preformatted text by 4 spaces

logstashadmin@logstashsapir:/usr/share/logstash/bin$ usermod -s /bin/bash logstash
usermod: no changes
logstashadmin@logstashsapir:/usr/share/logstash/bin$ su - logstash
Password:
indent preformatted text by 4 spaces

this is all i did till now
maybe you could see what i did wrong
indent preformatted text by 4 spaces

1 java -version
2 wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
3 sudo apt-get install apt-transport-https
4 echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
5 sudo apt-get update
6 sudo apt-get install logstash
7 sudo systemctl status logstash
8 sudo systemctl enable logstash
9 sudo systemctl start logstash
10 sudo lsof -i -P -n | grep logstash
11 sudo curl https://raw.githubusercontent.com/logzio/public-certificates/master/QuadCA.crt --create-dirs -o /usr/share/logstash/keys/TrustExternalCARoot.crt
12 ./logstash-plugin list | grep logstash-output-lumberjack
13 cd /usr/share/logstash/
14 ./logstash-plugin list | grep logstash-output-lumberjack
15 ./bin/logstash-plugin list | grep logstash-output-lumberjack
16 ./bin/logstash-plugin remove logstash-output-lumberjack
17 sudo ./bin/logstash-plugin remove logstash-output-lumberjack
18 sudo ./logstash-plugin install logstash-output-lumberjack-logzio
19 sudo ./bin/logstash-plugin install logstash-output-lumberjack-logzio
20 sudo ./bin/logstash-plugin install logstash-codec-sflow
21 sudo vi /etc/logstash/conf.d/sflowlog.conf
22 sudo systemctl stop logstash
23 sudo systemctl start logstash
24 sudo lsof -i -P -n | grep logstash
25 sudo systemctl stop logstash
26 cd /usr/share/logstash/bin
27 /logstash -f /etc/logstash/conf.d/sflowlog.conf
28 sudo /logstash -f /etc/logstash/conf.d/sflowlog.conf
29 ls
30 ./logstash -f /etc/logstash/conf.d/sflowlog.conf
31 sudo usermod -s /bin/bash logstash
32 sudo systemctl stop logstash
33 sudo systemctl start logstash
34 sudo systemctl status logstash
35 sudo lsof -i -P -n | grep logstash
36 su - logstash
37 sudo systemctl stop logstash
38 su - logstash -c ./logstash -f /etc/logstash/conf.d/sflowlog.conf
39 su - logstash -c './logstash -f /etc/logstash/conf.d/sflowlog.conf'
40 sudo chmod -R a+rwX /usr/share/logstash/data
41 ./logstash -f /etc/logstash/conf.d/sflowlog.conf
42 sudo ./logstash -f /etc/logstash/conf.d/sflowlog.conf
43 sudo systemctl stop logstash
44 sudo chmod -R ugo+rw /usr/share/logstash/keys/
45 ./logstash -f /etc/logstash/conf.d/sflowlog.conf
46 sudo chmod -R ugo+rw /usr/share/logstash/keys/TrustExternalCARoot.crt
47 ./logstash -f /etc/logstash/conf.d/sflowlog.conf
48 sudo systemctl stop logstash
49 ./usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
50 ./logstash "--path.settings" "/etc/logstash"
51 usermod -s /bin/bash logstash
52 su - logstash
53 su - logstash './usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"'

indent preformatted text by 4 spaces

this is my conf file

indent preformatted text by 4 spaces

"/etc/logstash/conf.d/sflowlog.conf" [readonly] 31L, 585C 1,1 All
input {
udp {
port => 6343
codec => sflow {}
add_field => {
"agent.hostname" => "sapir_sflow"
}
}
}
filter {

...

mutate {
add_field => { "token" => "**********" }
}

}
output {

stdout { codec => rubydebug }

lumberjack {
hosts => ["listener.logz.io"]
port => 5006
ssl_certificate => "/usr/share/logstash/keys/TrustExternalCARoot.crt"
codec => "json_lines"
}
}

#file {

path => "/home/logstashadmin/logstash.json"

codec => line { format => "json"}

}

#}

it’s asking for password because you run it as logstashadmin user. as i’ve mentioned, logstash user isn’t suppose to have shell access, this is merely to troubleshoot why you can’t run logstash as a service.

so login as root, switch user to logstash using
su - logstash
then run logstash from cli as logstash user

/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"