Filebeat - Error connecting to publish events

Elastic Stack Beats
1

Hi all. Trying to get a new ELK stack set up. I have two servers, one Linux with Filebeat running on it, and one Windows with Elasticsearch, Kibana, and X-Pack running on it. The servers are sitting on the same network segment with no firewalls or proxies between them. Here's my filebeat config:

filebeat.prospectors:

- input_type: log
  paths:
    - /var/log/httpd/error_log

- input_type: log
  paths:
    - /usr/local/redacted/logs/HttpLog
  include_lines: ["Execution"]

output.elasticsearch:
  hosts: ["xx.xx.xx.xx:82"]

  username: "filebeat"
  password: "redacted"

  index: "httpderrorlog-test-%{+yyyy.MM.dd}"
  indices:
    - index: "httplog-test-%{+yyyy.MM.dd}"
      when.contains:
        message: "Execution"

Here's the error I'm getting:

2017-10-24T15:21:41-04:00 ERR Connecting error publishing events (retrying): Get http://90.21.65.98:82: EOF
2017-10-24T15:21:41-04:00 DBG  send fail
2017-10-24T15:21:43-04:00 DBG  Flushing spooler because of timeout. Events flushed: 0
2017-10-24T15:21:45-04:00 DBG  End of file reached: /usr/local/redacted/logs/HttpLog; Backoff now.
2017-10-24T15:21:45-04:00 DBG  ES Ping(url=http://90.21.65.98:82, timeout=1m30s)
2017-10-24T15:21:45-04:00 DBG  Ping request failed with: Get http://90.21.65.98:82: EOF

I have the Elasticsearch server set up for HTTP on port 82 and Transport on 8081. I've verified that the user I specified in the yaml is created and has the proper roles in the cluster. I see no messages on the cluster whatsoever indicating that any traffic was received or authenticated whatsoever.

Does anybody have any suggestions?

2

Is the port configured using HTTP or HTTPS?

3

HTTP. I don't have SSL in place yet.

4

Looks like Elasticsearch is closing the connection.

Have you tried with curl http://90.21.65.98:82 ?

5

This is the response from the filebeat server:

curl: (52) Empty reply from server

I'm able to hit the cluster from Postman and Kibana on the same server as the cluster. I just talked with one of our network guys, and it seems most likely that the ES server (not Elasticsearch itself, but the underlying Windows server) is blocking the incoming connection.

EDIT: there is no software firewall on that server, Windows or otherwise. So that's not it either.

6

I figured it out. Seems it's always some tiny oversight that has a huge impact. In my x-pack filters, I forgot to include the IP range of the filebeat server for the HTTP filter.

xpack.security.enabled: true
xpack.security.http.filter.enabled: true
xpack.security.http.filter.allow: [ "xx.xx.xx.0/22", "yy.yy.yy.0/24", "zz.zz.zz.0/24" ]
xpack.security.http.filter.deny: _all
7

Thank you for sharing the solution!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.