MetricLogBeats To LogStash connection issue

ConorTier · August 10, 2018, 2:23pm

Hi,

Im having an issue with MetricLogBeats. Within metricbeat.log file I see the following: (IP address removed from logs for personal reasons)

2018-08-10T09:56:47.937-0400|ERROR|logstash/async.go:235|Failed to publish events caused by:     write tcp *IP*:62786->*IP*:5044: wsasend: An existing connection was forcibly closed by the remote host.|
|---|---|---|---|
|2018-08-10T09:56:48.939-0400|ERROR|pipeline/output.go:92|Failed to publish events: write tcp *IP*:62786->*IP*:5044: wsasend: An existing connection was forcibly closed by the remote host.|
|2018-08-10T09:56:51.943-0400|ERROR|pipeline/output.go:74|Failed to connect: dial tcp *IP*:5044: connectex: No connection could be made because the target machine actively refused it.|
|2018-08-10T09:56:56.962-0400|ERROR|pipeline/output.go:74|Failed to connect: dial tcp *IP*:5044: connectex: No connection could be made because the target machine actively refused it.|

I have the metricBeat.yml file practically left untouched bar : (ip of log stash server on a different machine to the metric beat server)

output.logstash:
    hosts: ["*IP*:5044"]

metricBeat module: system.yml is the only module enabled

- module: system
  period: 10s
  metricsets:
    - cpu
    #- load
    - memory
    - network
    - process
    - process_summary
  processes: ['.*']
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

#- module: system
#  metricsets:
#    - core
#    - diskio
#    - socket

- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

- module: system
  period: 15m
  metricsets:
    - uptime

#- module: system
#  period: 5m
#  metricsets:
#    - raid
#  raid.mount_point: '/'

When I start the MetricBeatService
Log stash seems to crash (continuously - depending if MetricBeat service is running or not) - giving the following error:

[2018-08-10T09:57:44,011][INFO ][logstash.pipeline        ] Pipeline main started
[2018-08-10T09:57:44,020][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-08-10T09:57:44,107][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-10T10:16:47,222][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"can't convert nil into String", "backtrace"=>["org/jruby/RubyString.java:4462:in `include?'", "(eval):188:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):186:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):129:in `filter_func'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}
[2018-08-10T10:16:47,420][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<TypeError: can't convert nil into String>, :backtrace=>["org/jruby/RubyString.java:4462:in `include?'", "(eval):188:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):186:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):129:in `filter_func'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}

C:\LogStash\logstash-5.6.3\bin>logstash -f logstash.config

The odd thing is when im on the Metricbeat server and i telnet into the LogStash server using port 5044 it seems to be successful.

I have seen similar discussions about this topic in with FileBeats but never any resolution.

Any Ideas?

adrisr · August 10, 2018, 3:21pm

This means there is an error in your Logstash pipeline configuration.

You seem to be accessing an array field that doesn't exist. You need to check for it to exists with an expression like if [field] and ...

Share your logstash configuration so we can help you further.

ConorTier · August 13, 2018, 7:53am

Hi @adrisr

Thanks for the followup.

app6 is the one giving issue, all my other apps seem to be running ok

app6 (metricbeat .yml has a tag in it defined app6 so I can perform the below conditions (write to a diff index)

# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["app6"]

Here is the contents of my logstash.config

# contents of logstash\bin\logstash.config

input {
beats {
port => 5044
}
}

filter {
if (("app1" in [tags]) or ([app] in "app2") or ([app] in "app3") or ([app] in "app4") or ([app] in "app5") or ([app] in "app6")) {
mutate {
strip => message
}

if (!("_grokparsefailure" in [tags])){
grok {
match => { message => "(?m)%{TIMESTAMP_ISO8601:sourceTimestamp} \[%{DATA:thread}\] %{LOGLEVEL:loglevel} %{DATA:logger} - %{GREEDYDATA:tempMessage}" }
}

mutate {
replace => [ "message" , "%{tempMessage}" ]
remove_field => [ "tempMessage", "sourceTimestamp" ]
}
}

date {
match => [ "sourceTimestamp", "YYYY-MM-dd HH:mm:ssZZ", "YYYY-MM-dd HH:mm:ssZ", "YYYY-MM-dd HH:mm:ss", "YYYY-MM-dd HH:mm:ss,SSSZZ", "YYYY-MM-dd HH:mm:ss,SSSZ", "YYYY-MM-dd HH:mm:ss,SSS", "YYYY-MM-dd HH:mm:ss:SSSZZ", "YYYY-MM-dd HH:mm:ss:SSSZ", "YYYY-MM-dd HH:mm:ss:SSS", "YYYY-MM-dd'T'HH:mm:ss,SSS", "ISO8601" ]
}
}
}

output {
if (!("_grokparsefailure" in [tags]) and (([app] in "app2") or ([app] in "app3") or ([app] in "app4") or ([app] in "app5"))){
elasticsearch {
hosts => "https://elasticUrl:9243"
user => "elastic"
password => "password"
index => "indexName1"
document_type => "logentry"
}
}

if (!("_grokparsefailure" in [tags]) and ("app6" in [tags])) {
elasticsearch {
hosts => "https://elasticUrl:9243"
user => "elastic"
password => "password"
index => "indexName2-%{+YYYY.MM.dd}"
document_type => "logentry"
}
}

if "app1" in [tags]{
elasticsearch {
hosts => "https://elasticUrl:9243"
user => "elastic"
password => "password"
index => "indexName3"
document_type => "logentry"
}
}
}

I have tried removing filters from logstash.config completely - still same issue,
Tried, removing filter + output - logstash seems to run without error, but at that point all its doing is listening and doing nothing else.
Tried, Leaving filter and removing output completely - still same issue

[FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<TypeError: can't convert nil into String>, :backtrace=>["org/jruby/RubyString.java:4462:in `include?'", "(eval):160:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):158:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):117:in `filter_func'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'", "C:/LogStash/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}

Im at a bit of a loss on this one

ConorTier · August 13, 2018, 8:04am

Same result if I exclude app6 from that if condition filter - and just do the date match - with the output to elasticsearch

adrisr · August 13, 2018, 9:59am

or ([app] in "app6")

I've been told that this can cause the error you're getting, if the app field is not set.

You should rewrite this as or (app and ([app] in "app6"))

Are you setting the app field in the events? Because I can only see you setting a tag with value app6.

ConorTier · August 13, 2018, 11:23am

When I write the condition as

if (("app1" in [tags]) or ([app] in "app2") or (app and ([app] in "app3")) or (app and ([app] in "app4")) or (app and ([app] in "app5")) or (app and ([app] in "app6"))) {
mutate {
strip => message
}

I get the following error:

[2018-08-13T06:53:38,678][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, ( at line 10, column 63 (byte 150) after filter {\nif (("app1" in [tags]) or ([app] in "app2") or (app "}

From my post earlier - I made mistake - the condtion within the filter should have been:

filter {
if (("app1" in [tags]) or ([app] in "app2") or ([app] in "app3") or ([app] in "app4") or ([app] in "app5") or ("app6" in [tags])) {
mutate {...

As you are correct app6 is not a tag and not a field.

But still the same issue.

In saying this, I just got it to work by adding :

fields:
  app: app6

To the metricbeat.yml

Looks like it was bombing out as app was not an existing field - shame the (app and ([app] in.. syntax did not work as then could check if exists

system · September 10, 2018, 11:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat Couldn't Send Logs to Logstash Beats filebeat	7	1098	September 23, 2019
I have installed a 7.17.3 metric beat and file beat, both the beats are unable to send data to the logstash Logstash	14	292	December 29, 2023
Metricbeat Communication problem Logstash	5	1561	April 25, 2018
Why Oh Why Oh Why will Metricbeat failing to connect to elasticsearch? Beats metricbeat	8	9606	July 24, 2017
Can Filebeat and Metricbeat together be using the same port of logstash Beats metricbeat	10	7248	August 30, 2017

MetricLogBeats To LogStash connection issue

Related topics