Logstach Starting issue!

Elastic Stack Logstash
1

Hi All,

I have configured File beat and logstach for log management .

I am able to start Filebeat but while starting getting below error ,

#####################################################

root@logstach_host bin]# ./logstash -f ../modules/test.conf
Sending Logstash's logs to /ELK/logstash-5.6.0/logs which is now configured via log4j2.properties
[2017-10-05T12:49:23,057][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/ELK/logstash-5.6.0/modules/fb_apache/configuration"}
[2017-10-05T12:49:23,061][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/ELK/logstash-5.6.0/modules/netflow/configuration"}
[2017-10-05T12:49:23,354][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-10-05T12:49:23,939][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"filebeat_host:5044"}
[2017-10-05T12:49:24,031][INFO ][logstash.pipeline ] Pipeline main started
[2017-10-05T12:49:24,120][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2017-10-05T12:49:24,189][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-10-05T12:49:30,742][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Beats port=>5044, host=>"filebeat_host", id=>"4101285bfc7046061e79ac78976033137330de0d-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_acf1f1ef-689f-4e36-a8b8-4017185476e4", enable_metric=>true, charset=>"UTF-8">, ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60>
Error: Cannot assign requested address
[2017-10-05T12:49:31,744][INFO ][org.logstash.beats.Server] Starting server on port: 5044
Oct 05, 2017 12:49:31 PM io.netty.channel.AbstractChannel$AbstractUnsafe register
WARNING: Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x7f8152f2]
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:840)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:342)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:335)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:765)
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:85)
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:330)
at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:281)
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:277)
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:259)
at org.logstash.beats.Server.listen(Server.java:69)

######################################################

[2017-10-05T12:49:30,742][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Beats port=>5044, host=>"filebeat_host", id=>"4101285bfc7046061e79ac78976033137330de0d-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_acf1f1ef-689f-4e36-a8b8-4017185476e4", enable_metric=>true, charset=>"UTF-8">, ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60>
Error: Cannot assign requested address
[2017-10-05T12:49:31,744][INFO ][org.logstash.beats.Server] Starting server on port: 5044

Which plugin it is referring to also at filebeat side also facing some issues ,

}
2017/10/05 10:49:25.144884 output.go:109: DBG output worker: publish 1997 events
2017/10/05 10:49:25.146132 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 10:49:26.147343 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 10:49:28.148827 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 10:49:32.150197 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 10:49:40.151842 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 10:49:50.355352 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=18 filebeat.harvester.running=18 filebeat.harvester.started=18 libbeat.publisher.published_events=1997
2017/10/05 10:49:56.153284 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 10:50:20.355302 metrics.go:34: INFO No non-zero metrics in the last 30s
^C2017/10/05 10:50:21.802480 filebeat.go:267: INFO Stopping filebeat

Can someone please help on this same .

2

Which JVM are you using?

3

Hi ,

I am using JAVA 1.8 .

[root@vielk01dsy bin]# java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)

4

Is this the host Logstash is running on? Can you try leaving this out so it uses the default (0.0.0.0)?

5

no it is filebeat host .

logstatch is on different host .

6

Then remove it as it specified which host Logstash should bind to.

7

Sorry didnt get ,

if i remove this host then how logstach would receive log from filebeat ?

please correct if i am wrong .

8

You should not specify host parameter for the beats input. Using the default will make Logstash bind to all network interfaces on the host it is running, which will allow Filebeat to connect to it.

9

By removing host parameter ,

i am facing errors with different way ,

after starting loghstach

./logstash -f ../modules/test.conf

[root@vielk01dsy bin]# ./logstash -f ../modules/test.conf
Sending Logstash's logs to /ELK/logstash-5.6.0/logs which is now configured via log4j2.properties
[2017-10-05T14:10:38,836][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/ELK/logstash-5.6.0/modules/fb_apache/configuration"}
[2017-10-05T14:10:38,841][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/ELK/logstash-5.6.0/modules/netflow/configuration"}
[2017-10-05T14:10:39,155][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-10-05T14:10:40,046][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2017-10-05T14:10:40,123][INFO ][logstash.pipeline ] Pipeline main started
[2017-10-05T14:10:40,142][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2017-10-05T14:10:40,252][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

it is not going beyond this .

Also on Filebeat_host i am getting below errors ,

sudo ./filebeat -e -c filebeat.yml -d "publish"

2017/10/05 12:11:52.447634 output.go:109: DBG output worker: publish 2006 events
2017/10/05 12:11:52.448919 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 12:11:53.450350 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 12:11:55.451680 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 12:11:59.452982 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host
2017/10/05 12:12:07.454370 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 10.81.104.105:5044: getsockopt: no route to host

#########

please suggest now .

Filebeat config file :-

filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • input_type: log

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /var/log/*.log
      #- c:\programdata\elasticsearch\logs*

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["logstach_host:5044"]

Logstach config file :-

input {
beats {
port => "5044"
}
}

The filter part of this file is commented out to indicate that it is

optional.

filter {

}

output {
stdout { codec => rubydebug }
}

10

Please format your config file entries as preformatted text when pasting them as it is very hard to read them as they are currently formatted.

Do you have connectivity between the nodes? Can you e.g. telnet to port 5044 on the Logstash host from the node where Filebeat is running?

11

Filebeat config file :-

filebeat.prospectors:
input_type: log
paths:
/var/log/.log
#- c:\programdata\elasticsearch\logs

#----------------------------- Logstash output --------------------------------
output.logstash:
The Logstash hosts

hosts: ["logstach_host:5044"]

Logstach config file :-

input {
beats {
port => "5044"
}
}
filter {
}

output {
stdout { codec => rubydebug }
}

12

I have shared config files above .

also i cant telenet to logstach server

telnet logstach_host 5044
Trying logstach_host...
telnet: connect to address logstach_host : No route to host

how to resolve it .

do i need to update iptables ?

13

If you can not telnet and the hostname is correct, you may need to check iptables etc.

14

Need your help on Implementing ELK stack with architecture .

DO you have any docs for the same

15

This blog post may be useful:

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.