I am trying to get ElasticSearch setup on my Ubuntu server 18 on AWS. logstash seems to be installed: "logstash/stable,now 1:7.5.1-1 all [installed]"
But when I check the config of logstash, it seems like logstash is not installed:
"service logstash configtest
logstash: unrecognized service"
I have been using this guide: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04.
I personally do not do conf checks this way
/usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
if your not using /etc/logstash for your configuration files alter the path
I am not sure if this is normal, but when I run that, I get a lot of errors about permissions. The owner user and group was logstash. And these were the permissions: drwxrwxr-x. The errors looked like this: "An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/dead_letter_queue" must be a writable directory. It is not writable." I chmod 777 them, which fixed it, but that does not seem like a smart security-wise. Any ideas why it can't write to it?
I also seem to be getting a lot of Java output/errors:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
2020-01-05 14:00:25,376 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:237)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:158)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:640)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:44)
at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:16)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:239)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:369)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:260)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:615)
at org.apache.logging.log4j.core.LoggerContext.setConfigLocation(LoggerContext.java:605)
at org.logstash.log.LoggerExt.reconfigure(LoggerExt.java:160)
at org.logstash.log.LoggerExt$INVOKER$s$1$0$reconfigure.call(LoggerExt$INVOKER$s$1$0$reconfigure.gen)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:833)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)
at usr.share.logstash.logstash_minus_core.lib.logstash.runner.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:263)
at usr.share.logstash.logstash_minus_core.lib.logstash.runner.RUBY$method$execute$0$VARARGS(/usr/share/logstash/logstash-core/lib/logstash/runner.rb)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_0_dot_6_dot_5.lib.clamp.command.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_0_dot_6_dot_5.lib.clamp.command.RUBY$method$run$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)
at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1156)
at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1143)
at org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)
at usr.share.logstash.logstash_minus_core.lib.logstash.runner.RUBY$method$run$0(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242)
at usr.share.logstash.logstash_minus_core.lib.logstash.runner.RUBY$method$run$0$VARARGS(/usr/share/logstash/logstash-core/lib/logstash/runner.rb)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_0_dot_6_dot_5.lib.clamp.command.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)
at usr.share.logstash.lib.bootstrap.environment.RUBY$script(/usr/share/logstash/lib/bootstrap/environment.rb:73)
at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
at org.jruby.ir.Compiler$1.load(Compiler.java:89)
at org.jruby.Ruby.runScript(Ruby.java:854)
at org.jruby.Ruby.runNormally(Ruby.java:777)
at org.jruby.Ruby.runNormally(Ruby.java:795)
at org.jruby.Ruby.runFromMain(Ruby.java:607)
at org.logstash.Logstash.run(Logstash.java:113)
at org.logstash.Logstash.main(Logstash.java:47)
2020-01-05 14:00:25,381 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@1c595228] unable to create manager for [/var/log/logstash/logstash-plain.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@1e64ee13[pattern=/var/log/logstash/logstash-plain-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)]), strategy=DefaultRolloverStrategy(min=1, max=30, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]} %m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@1c595228] unable to create manager for [/var/log/logstash/logstash-plain.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@1e64ee13[pattern=/var/log/logstash/logstash-plain-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)]), strategy=DefaultRolloverStrategy(min=1, max=30, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]} %m%n, filePermissions=null, fileOwner=null]]
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
...
BUMP
Here's the updated DigitalOcean guide for Ubuntu 18.04.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.