Filebeeat sending log, but logstash not running, connection refused!

Elastic Stack Logstash
1

hi
im used filebeat to many server, shipped nginx log to logstash
in this time and months my elk server is very good worked
but, my 1 line added grok pattern to syslog-filter.conf, and restart logstash ,,,
my elk and Concerning Logstash not worked
this is wehn, my elasticsearch and logatash and kibana ... services this up and enable and active!
but ...
my nginx servers ...
telnet to 5044
and
telnet to 5443
connection refused

this log, 1 server Logs (filebeat logs)

> 2018-01-23T10:21:21+03:30 ERR  Failed to connect: dial tcp 172.17.11.202:5443: getsockopt: connection refused
> 2018-01-23T10:21:28+03:30 INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=11769216 beat.memstats.memory_alloc=5935656 beat.memstats.memory_total=73881024 filebeat.harvester.open_files=5 filebeat.harvester.running=6 libbeat.config.module.running=0 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=2048 registrar.states.current=35
> 2018-01-23T10:47:16+03:30 INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=11297792 beat.memstats.memory_alloc=5872352 beat.memstats.memory_total=26557112 filebeat.harvester.open_files=5 filebeat.harvester.running=6 libbeat.config.module.running=0 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 registrar.states.current=37
> 2018-01-23T10:47:22+03:30 ERR  Failed to connect: dial tcp 172.17.11.202:5443: getsockopt: connection refused
> 2018-01-23T10:47:46+03:30 INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=11297792 beat.memstats.memory_alloc=6012704 beat.memstats.memory_total=26697464 filebeat.harvester.open_files=5 filebeat.harvester.running=6 libbeat.config.module.running=0 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=2048 registrar.states.current=37
> 2018-01-23T14:22:45+03:30 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=11490800 beat.memstats.memory_alloc=5802160 beat.memstats.memory_total=153496216 filebeat.harvester.open_files=3 filebeat.harvester.running=2 libbeat.config.module.running=0 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 registrar.states.current=3
2

Maybe more log information for Logstash is useful for community.:grinning:

3

Which logstash information do I give?

4

You need to confirm your logstash service is up.

But per your filebeat log, connection refused should represent logstash is inactivate.

5 
[2018-01-24T10:42:41,061][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.1.0"}
[2018-01-24T10:42:41,260][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-01-24T10:42:41,390][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, \", ', -, [, { at line 18, column 7 (byte 1393) after filter {\n  grok {\n    match => { message => [\n      \"%{IP:client} %{HTTPDUSER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp_server_genaration}\\] \\\"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\\\" %{NUMBER:status_code} %{NUMBER:bytes} %{QS:refferer} %{QS:user_agent} length %{NUMBER:length} rtime %{NUMBER:request_time} uri %{URIPATHPARAM:uri} realip %{IP:realip}\",\n      \"%{IP:client} %{HTTPDUSER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp_server_genaration}\\] \\\"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\\\" %{NUMBER:status_code} %{NUMBER:bytes} %{QS:refferer} %{QS:user_agent} length %{NUMBER:length} rtime %{NUMBER:request_time} uri %{URIPATHPARAM:uri}\",\n      \"(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \\[%{LOGLEVEL:severity}\\] %{POSINT:pid}#%{NUMBER:threadid}\\: \\*%{NUMBER:connectionid} %{GREEDYDATA:errormessage}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: %{GREEDYDATA:request}\",\n      \"%{IP:client} %{HTTPDUSER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp_server_genaration}\\] \\\"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\\\" %{NUMBER:status_code} %{NUMBER:bytes} %{QS:refferer} %{QS:user_agent}\",\n      ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[2018-01-24T10:42:54,443][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-01-24T10:42:54,446][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-01-24T10:42:54,750][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

this log logstash log
my logstash service is fully up
but no listen port 5443 or 5044 !!!

6

No. Configuration error.

[2018-01-24T10:42:41,390][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, ", ', -, [, { at line 18, column 7 (byte 1393) after filter {\n grok {\n match => { message => [\n "%{IP:client} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp_server_genaration}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:status_code} %{NUMBER:bytes} %{QS:refferer} %{QS:user_agent} length %{NUMBER:length} rtime %{NUMBER:request_time} uri %{URIPATHPARAM:uri} realip %{IP:realip}",\n "%{IP:client} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp_server_genaration}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:status_code} %{NUMBER:bytes} %{QS:refferer} %{QS:user_agent} length %{NUMBER:length} rtime %{NUMBER:request_time} uri %{URIPATHPARAM:uri}",\n "(?%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:errormessage}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: %{GREEDYDATA:request}",\n "%{IP:client} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp_server_genaration}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:status_code} %{NUMBER:bytes} %{QS:refferer} %{QS:user_agent}",\n ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

7

im elk services, During this time he worked without problems
im added 1 line grok pattern, logstash DOWN!
why ?

8

Logstash receives message from filebeat while send format json to es. Kibana will load data from es. So even you see kibana/es working, not demonstrate logstash working.
You need to check config of logstash.

9

Thnks,
im checked logstash.yml or Inside the folder conf.d ??

10

Thanks Thanks
im finding to problem ...
Inside the folder conf.d ...
my grok pattern ... conflicted
2 grok conflicted .
deleted last lined added, fix .

11

Good job.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.