Hello all,
I'm new to elastic stack and i'm seeking some help.
The log shipping worked just fine when i had 2 nodes but the second i'v added a third one i had this error when running the config file
help would be much appreciated.
Hi Karto,
Can you pls explain a bit more and also share the filebeat.yml and logstash.yml config here so that i can check and identify the problem.
Thanks,
Harsh Bajaj
Thank you for relying,
So i have one node containing my elk server (logstash 5.6 kibana 5.6and elasticsearch5.6)
And i'm trying to ship logs from remote servers where filebeat 5.6 is installed.
The shipping worked fine when i tested with one node but when i tried to add another node i had that error
Running configuration with file input works.
filebeat.yml
https://pastebin.com/nVhcZtu6
logstash.yml
https://pastebin.com/zYiSf5xz
This look like a bug in the logstash beats input, can you run the following command bin/logstash-plugin list --verbose beats from your logstash node and add the output to this issue?
Here's what i get
logstash-input-beats (3.1.26)
Hi karto,
i need logstash pipeline conf file o/p here. please share.
/etc/logstash/conf.d/
Hello Harsh
Yes here's my config
https://pastebin.com/ySEcajS5
Hi @karto,
I chceked your config file and found that you have mentioned "host" parameter in "beats" plugin is :
host=>"0.0.0.0"
Please comment this and start filebeat and let me know if there is any error.
Thanks,
Harsh Bajaj
Hello @harshbajaj16,
Unfortunately i still have the same error after commenting that line.
could you please paste your error here ?
because in your previous file there were many errors:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-04-11 10:05:50.847 [[main]<beats] Server - Starting server on port: 5044
[ERROR] 2018-04-11 10:05:57.122 [[main]<beats] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Beats port=>5044, host=>"0.0.0.0", ssl=>true, ssl_certificate=>"/etc/pki/tls/certs/logstash-forwarder.crt", ssl_key=>"/etc/pki/tls/private/logstash-forwarder.key", id=>"8f47c33f5ada3a1471fe3b7dae07f6cb7358e76e-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_f9eff4c6-afdb-4116-bffd-1084dc08348a", enable_metric=>true, charset=>"UTF-8">, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>8>
Error: Address already in use
[INFO ] 2018-04-11 10:05:58.123 [[main]<beats] Server - Starting server on port: 5044
[WARN ] 2018-04-11 10:05:58.125 [[main]<beats] AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x5606c0e8]
could you please confirm that you are getting same error or diff?
Yes It's the same error
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-04-12 09:02:06.562 [[main]<beats] Server - Starting server on port: 5044
[ERROR] 2018-04-12 09:02:12.843 [[main]<beats] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Beats port=>5044, ssl=>true, ssl_certificate=>"/etc/pki/tls/certs/logstash-forwarder.crt", ssl_key=>"/etc/pki/tls/private/logstash-forwarder.key", id=>"35f68e6919d41d92df9a1b0e31802b31b181a0c5-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_032fbd3f-f8a9-4ced-80dc-4ef277ec3522", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>8>
Error: Address already in use
[INFO ] 2018-04-12 09:02:13.844 [[main]<beats] Server - Starting server on port: 5044
[WARN ] 2018-04-12 09:02:13.846 [[main]<beats] AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x861537bc]
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:840) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:342) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:335) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:765) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:85) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:330) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:281) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:277) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:259) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at org.logstash.beats.Server.listen(Server.java:68) [logstash-input-beats-3.1.26.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:436) [?:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:300) [?:?]
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:35) [?:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134) [?:?]
at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60) [?:?]
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105) [?:?]
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) [?:?]
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) [?:?]
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182) [?:?]
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203) [?:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168) [?:?]
at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57) [?:?]
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105) [?:?]
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) [?:?]
at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221) [?:?]
at org.jruby.ast.RescueNode.interpret(RescueNode.java:116) [?:?]
at org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96) [?:?]
at org.jruby.ast.BeginNode.interpret(BeginNode.java:83) [?:?]
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105) [?:?]
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) [?:?]
you r are getting this error while starting logstash or filebeat ?
please confirm.
While running the beat config in the server containing logstash.
logstash doesnt show errors and but when debugging filebeat i get
2018/04/12 09:47:47.600105 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 172.31.39.192:5044: getsockopt: connection refused 2018/04/12 09:47:48.601441 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 172.31.39.192:5044: getsockopt: connection refused 2018/04/12 09:47:50.602458 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 172.31.39.192:5044: getsockopt: connection refused
Hi @karto,
it means your logstash is not working properly. please check your logstash logs.
Hello @harshbajaj16
After further digging i'v found this error in logstash log
[FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
I think that i'm missing some configuration
What i'm trying to do is to centralise logs from multiple 'client machines', for that i'v installed the ELK stack on a server machine and filebeat on each client machine.
Filebeat configuration is the same for each and every client machine sending data on port 5044
and i have only one configuration file.
I want to know if there is more configuration to do in order for logstash to to recieve logs from multiple machines.
Thanks
This error message is showing you are running two logstash on same server thats ok you can run but data directory should be diff. go to the YML file and change the data directory to define new path in "path.data".
Thanks,
Harsh Bajaj
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.