Hi,
Im having an issue with MetricLogBeats. Within metricbeat.log file I see the following: (IP address removed from logs for personal reasons)
2018-08-10T09:56:47.937-0400|ERROR|logstash/async.go:235|Failed to publish events caused by: write tcp *IP*:62786->*IP*:5044: wsasend: An existing connection was forcibly closed by the remote host.|
|---|---|---|---|
|2018-08-10T09:56:48.939-0400|ERROR|pipeline/output.go:92|Failed to publish events: write tcp *IP*:62786->*IP*:5044: wsasend: An existing connection was forcibly closed by the remote host.|
|2018-08-10T09:56:51.943-0400|ERROR|pipeline/output.go:74|Failed to connect: dial tcp *IP*:5044: connectex: No connection could be made because the target machine actively refused it.|
|2018-08-10T09:56:56.962-0400|ERROR|pipeline/output.go:74|Failed to connect: dial tcp *IP*:5044: connectex: No connection could be made because the target machine actively refused it.|
I have the metricBeat.yml file practically left untouched bar : (ip of log stash server on a different machine to the metric beat server)
output.logstash:
hosts: ["*IP*:5044"]
metricBeat module: system.yml is the only module enabled
- module: system
period: 10s
metricsets:
- cpu
#- load
- memory
- network
- process
- process_summary
processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
#- module: system
# metricsets:
# - core
# - diskio
# - socket
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
- module: system
period: 15m
metricsets:
- uptime
#- module: system
# period: 5m
# metricsets:
# - raid
# raid.mount_point: '/'
When I start the MetricBeatService
Log stash seems to crash (continuously - depending if MetricBeat service is running or not) - giving the following error:
[2018-08-10T09:57:44,011][INFO ][logstash.pipeline ] Pipeline main started
[2018-08-10T09:57:44,020][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-10T09:57:44,107][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-10T10:16:47,222][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"can't convert nil into String", "backtrace"=>["org/jruby/RubyString.java:4462:in `include?'", "(eval):188:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):186:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):129:in `filter_func'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}
[2018-08-10T10:16:47,420][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<TypeError: can't convert nil into String>, :backtrace=>["org/jruby/RubyString.java:4462:in `include?'", "(eval):188:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):186:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):129:in `filter_func'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}
C:\LogStash\logstash-5.6.3\bin>logstash -f logstash.config
The odd thing is when im on the Metricbeat server and i telnet into the LogStash server using port 5044 it seems to be successful.
I have seen similar discussions about this topic in with FileBeats but never any resolution.
Any Ideas?