Filebeat 6.3.2 / Logstash 6.3.2

Elastic Stack Beats
1

Hello !

Today i've tried to make filebeat (that is on one of my servers) send logs to another server, that have logstash, elasticsearch and kibana.

I've got some problems and one in particular that i can't find a solution, here the logs of it :

ERROR	logstash/async.go:235	Failed to publish events caused by: write tcp 192.168.250.88:50022->192.168.250.224:5044: write: connection reset by peer

I've seen some posts with the same problem, and some say that updating a plugin will solve this, but i never installed any plugin, or is there included plugins when installing logstash ?
So yet i can't resolve this, look forward for your help :slightly_smiling_face:

2

Each input/filter/output is a plugin to Logstash. Many are bundled with Logstash (e.g. logstash-input-beats). They updatable independent of the main Logstash install. See Updating the Beats Input Plugin for Logstash.

Have you testing the connection via filebeat test output?

Or tested it via telnet?

If neither of those tests are able to connected then log onto the Logstash server and verify that it is listening with a command like sudo netstat -anp | grep 5033.

3

The Filebeat test output is ok, but telnet do not look ok, can't connect, tried why telnet myip and telnet myip 5044 , the first don't work and with 5044, when i type something it close the connection (i use iptables on my ELK server, and i authorize ports 22, 9200, 5601 and 5044, is there an other one to authorize for filebeat ?

When doing sudo netstat -anp | grep 5033 no output.

Also, in my filebeat log, sometime there is :
ERROR logstash/async.go:235 Failed to publish events caused by: write tcp IPclient:41148->IPserver:5044: write: connection reset by peer
ERROR pipeline/output.go:92 Failed to publish events: write tcp IPclient:41148->IPserver:5044: write: connection reset by peer

(Sorry for the late answer, was working on something else)

4

How about with 5044? :slight_smile: my bad

But if filebeat test output is passing and telnet <ip> 5044 connects (it's normal to have it disconnect you after you type something because it rejects that input and resets the connection) then it seems like connection is being established OK.

So the next step is to inspect the Logstash logs (start LS with -debug) and see if there are any errors while Filebeat is sending data. It would be useful if you shared the LS configuration that you are using too.

Another test you can do is to disable all outputs in your Logstash config and add only a single stdout output. This way the outputs cannot block the pipeline.

output { 
  stdout { 
    codec  => rubydebug {
      metadata => true
    }
  }
}
5

Okay :slight_smile: i'll do that tomorrow and tell you the results :slight_smile:

6

So sudo netstat -anp | grep 5033 is ok.

Now i can't find anything like bin/logstash, there is a dir /opt/logstash/bin, but nothing in there, the only one that seem like that is in /usr/share, does it matter ? Or i need to install something ?

For the configuration : (in /etc/logstash/conf.d/logstash.conf)

input {
    beats {
           	port => 5044
            client_inactivity_timeout => 3000
    }
}

output {
    elasticsearch {
            hosts => "myipserver:9200"
            manage_template => false
            index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
            document_type => "%{[@metadata][type]}"
    }
}

There's also a /etc/logstash/logstash.yml, isnt that the conf file or its something else ?

7

The location depends on how it was installed. The paths are documented here.

8

Your config deviates from what we recommend. Specifically the index name differs. We include the version there an the default index template only apply to indexes containing a version.

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

And since you are using Logstash, make sure that you installed the index template to Elasticsearch.

9

Actually i don't see any error when sending directly to elasticsearch, just when sending to logstash, just many Non-zero metrics in the last 30s.
Anyway, i can't figure out why my ELK server, on Kibana, don't receive logs (he receive, but in discover i can see like 1 count per day), maybe my tested servers are just not active enought ?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.