Logstash stopping and restatŕting every 5 seconds and is using more than 600% of cpu

Elastic Stack Logstash
1

Logstash is using ore than 600% cpu and is starting and stopping again, actually it is working fine for the filters but still the cpu issue is critical and i am getting below error in the logstash logs -

[2020-07-30T10:27:15,100][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2020-07-30T10:27:38,009][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:528:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:288:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:199:in `block in validate_all'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:198:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:305:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:263:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in `<main>'"]}
[2020-07-30T10:27:38,169][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2020-07-30T10:28:01,168][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:528:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:288:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:199:in `block in validate_all'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:198:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:305:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:263:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in `<main>'"]}
[2020-07-30T10:28:01,327][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

it would be helpful if someone could help with this issue

2

Please share your configuration file. Do you have any configuration file in conf.d?

or check your configuration file in this way and show message:

/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/

d

3

Did you make sure that Logsash has writing permissions for this directory?

4

Hi @d.silwon the conf.d is an emty folder now but i am using the below configuration file for my data parsing -


 input {
    beats {
        port => "5044"
    }
}
filter {
    
        dissect { mapping => { "message" => "%{timestamp1->} %{+timestamp1} %{+timestamp1} %{Theatre} %{Product} %{Severity} %{Body}" 

"Body" => "%{1} %{2} %{3} %{4} %{5} %{6} %{7}" } }
       

        kv { source => "Body" }
    }
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

please check

5

The conf file looks good. Please check what is suggested by @Jenni
I have:
drwxr-x---. 2 logstash logstash 6 Jun 25 07:37 /usr/share/logstash/data/queue

6

Hi @Jenni i tried making it writable but same issues, can you please suggest how i can give permissions to logstash for this

7

Hi @d.silwon is this the command i can use to give permission?

8

Show rights:
ls -ld /usr/share/logstash/data/queue

The rights you can change in this way:
chmod 750 /usr/share/logstash/data/queue
chown logstash.logstash /usr/share/logstash/data/queue

but check it first.

9

it is showing the rigts as below -

drwxrwxr-x 2 root root 4096 Jun 16 14:18 /usr/share/logstash/data/queue

should i change them?

10

I think you should change rights for owner and group, should be logstash logstash

11

i changed it, now i am receiving the below error-

[2020-07-30T13:08:28,542][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>java.nio.file.AccessDeniedException: /usr/share/logstash/data/.lock, :backtrace=>["sun.nio.fs.UnixException.translateToIOException(sun/nio/fs/UnixException.java:84)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:102)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:107)", "sun.nio.fs.UnixFileSystemProvider.newFileChannel(sun/nio/fs/UnixFileSystemProvider.java:177)", "java.nio.channels.FileChannel.open(java/nio/channels/FileChannel.java:287)", "java.nio.channels.FileChannel.open(java/nio/channels/FileChannel.java:335)", "org.logstash.FileLockFactory.obtainLock(org/logstash/FileLockFactory.java:89)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:471)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:373)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:377)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_0_dot_6_dot_5.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_0_dot_6_dot_5.lib.clamp.command.RUBY$method$run$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/clamp_minus_0_dot_6_dot_5/lib/clamp//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.runner.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:263)", "usr.share.logstash.logstash_minus_core.lib.logstash.runner.RUBY$method$run$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/runner.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_0_dot_6_dot_5.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132)", "usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:88)", "java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)", "org.jruby.Ruby.runScript(org/jruby/Ruby.java:1205)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1128)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1146)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:958)", "org.logstash.Logstash.run(org/logstash/Logstash.java:133)", "org.logstash.Logstash.main(org/logstash/Logstash.java:67)"]}
[2020-07-30T13:08:28,706][ERROR][org.logs
12

Did you change the permissions recursively or only for the top directory?

13

yes i changed for another directory as well after that it was asking for this directory /usr/share/logstash/data/dead_letter_queue
to be writable

14

But did you give it permissions for /usr/share/logstash/data/.lock?

15

no i gave permissions for /usr/share/logstash/data/queue

should i give permissions to /usr/share/logstash/data/.lock ?

and using these comaands ?-

chmod 750 /usr/share/logstash/data/.lock
chown logstash.logstash /usr/share/logstash/data/.lock

16

Please do it in this way:
cd /usr/share
chown -R logstash.logstash logstash

1 Like
17

i tried this i think the issue is still there, i get the below messages

[2020-08-19T08:45:11,922][INFO ][org.logstash.beats.Server][main][a69701381b84e5bb13a99bcc92b9d3f1af97f5ac476344c6ab73bc10e6944c57] Starting server on port: 5044
[2020-08-19T08:45:18,142][ERROR][logstash.javapipeline    ][main][a69701381b84e5bb13a99bcc92b9d3f1af97f5ac476344c6ab73bc10e6944c57] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats port=>5044, id=>"a69701381b84e5bb13a99bcc92b9d3f1af97f5ac476344c6ab73bc10e6944c57", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_df7eaa0d-08b5-4237-9d91-34d05dab2627", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>16>
  Error: Address already in use
  Exception: Java::JavaNet::BindException
  Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:433)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:425)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:220)
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:130)
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:558)
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1358)
io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:501)
io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:486)
io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:1019)
io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:254)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:366)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:163)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:404)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:462)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:897)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:748)
[2020-08-19T08:45:19,144][INFO ][org.logstash.beats.Server][main][a69701381b84e5bb13a99bcc92b9d3f1af97f5ac476344c6ab73bc10e6944c57] Starting server on port: 5044
18

Hi,

But the error is different right now. Please solve it.

Regrads,
Daniel

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.