Can't get filebeat to work properly on any port other than 9200

[rdesanno (Rob)]

I have a problem that is driving me crazy. I set up a POC for ELK and configured filebeat to send our logs via loghost:9200, the same port that logstash is listening to. That was fine for the test but now I want to use grok and start redirecting filebeat to loghost:5044 and then to loghost:9200. Easy to do configuration-wise, but it's not working as I think it should.

:5044 is open on my loghost and I can telnet in from another host or even from the loghost itself to post test messages but when it comes to logs, nothing really gets sent. I do see some messages like "User-Agent: Go-http-client/1.1" or "GET / HTTP/1.1" in Kibana, but the filebeat log from any host always returns

ERR Connecting error publishing events (retrying): Get http://loghost:5044: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Changing /etc/filebeat/filebeat.yml to use port :9200 makes everything right in the world again.

Seeing that this happens from my loghost directly, i’m pretty confident that it’s a configuration issue on my part but can’t think of what i'm not seeing.

As an aside, I have a few other exposed ports on my loghost that I use for remote syslogging and that seems to be working well. Pointing filebeat to one of those ports gives me the same errors.

firewalld is down, iptables is flushed, selinux is disabled, ulimit is unlimited.

For these examples, loghost is the name of my ELK server.

# telnet loghost 5044
Trying xx.xx.xx.xx...
Connected to loghost.
Escape character is '^]'.
test2
^]
telnet> quit
Connection closed.

# lsof -i:5044
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    30131 logstash   49u  IPv4 366055      0t0  TCP *:lxi-evntsvc (LISTEN)

grep -v ^# /etc/logstash/logstash.yml
node.name: loghost
path.data: /var/lib/logstash
pipeline.workers: 16
pipeline.output.workers: 8
path.config: /etc/logstash/conf.d
config.reload.automatic: true
config.reload.interval: 30
http.host: "loghost”
path.logs: /var/log/logstash

# cat /etc/logstash/conf.d/filebeat.conf
input {
        tcp {
                port => 5044
        }
}

filter {
	grok {
		add_tag => [ "filebeat" ]
		break_on_match => true
		patterns_dir => [ "/etc/logstash/patterns" ]
                match => [ "message", "%{SYSLOGBASE} %{GREEDYDATA}"]
	}
}

output {
		elasticsearch {
		hosts => "loghost:9200"
		index => "filebeat-%{+YYYY.MM.dd}"
	}
}

[Blaxican707 (Ray (Blaxican707))]

Shouldn't "loghost" be "localhost" ?

Most likely there is something wrong with your .conf file that when filebeat sends its logs, you receive a connection refused. You can run a netstat and see what else is binded to that port, but more than likely, there is something wrong with either your filebeat.yml, or configure issues (probably fat fingured) in your .conf file.

[byakuinss]

Have you tried to use beats input plugin?
I use this config to collect log files from filebeat, then output to elasticsearch. It works fine to me.

input {
  beats {
    port => 5044
  }
}

[Blaxican707 (Ray (Blaxican707))]

Upon further review, the reason 5044 is not answering your filebeat is because there is something wrong with your logstash.conf (or whatever you named it too) file. The reason why 9200 works is because elasticsearch is receiving and is not filtering/parsing the data such as logstash does. You can telnet to 5044 all day, but if you try to send data to logstash, it'll refuse the connection because the logstash conf file has been written incorrectly.

your input should look like for starters, you have to specify beats as the input method...

input
{
beats {
port => 5044
}
}

[steffens (Steffen Siering)]

Can you share your beats configuration. beats->logstash is not HTTP. The fact the error message complains about HTTP makes me wonder if you configured the elasticsearch output to send to logstash.

[rdesanno (Rob)]

Thanks for the suggestion about installing the beats plugin on loghost. Did that, restarted logstash, and added beats to /etc/logstash/conf.d/filebeat.conf, but no dice.

input {
        beats {
                port => 5044
        }
}

[2017-09-06T10:26:11,071][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}

Here's what my /etc/filebeat/filebeat.yml looks like:

#=========================== Filebeat prospectors =============================

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/cron
    - /var/log/dmesg
    - /var/log/dracut.log
    - /var/log/maillog
    - /var/log/messages
    - /var/log/secure
    - /var/log/yum.log
  tags: ["syslog"]
  ignore_older: 24h
  exclude_lines: ["UDP"]
  exclude_files: [".gz$",".txt$"]

#================================ General =====================================

tags: ["test"]

fields:
  env: test
  os: 7
  loc: nj

#================================ Outputs =====================================
output.elasticsearch:

  hosts: ["loghost:5044"]

[rdesanno (Rob)]

Some more information:

/etc/logstash/logstash.yml:
grep -v ^# logstash.yml
node.name: loghost
path.data: /var/lib/logstash
pipeline.workers: 16
pipeline.output.workers: 8
path.config: /etc/logstash/conf.d
config.reload.automatic: true
config.reload.interval: 30
http.host: "loghost"
http.port: 9600
path.logs: /var/log/logstash

Sample from /var/log/filebeat/filebeat log on my test host:

2017-09-06T10:36:12-04:00 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_errors=1 libbeat.es.publish.write_bytes=127
2017-09-06T10:36:42-04:00 INFO No non-zero metrics in the last 30s
2017-09-06T10:36:45-04:00 ERR Connecting error publishing events (retrying): Get http://loghost:5044: read tcp testmachine.ip.addr:51835->loghost.ip.addr:5044: read: connection reset by peer
2017-09-06T10:37:12-04:00 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_errors=1 libbeat.es.publish.write_bytes=127
2017-09-06T10:37:42-04:00 INFO No non-zero metrics in the last 30s
2017-09-06T10:37:45-04:00 ERR Connecting error publishing events (retrying): Get http://loghost:5044: read tcp testmachine.ip.addr:54609->loghost.ip.addr:5044: read: connection reset by peer
2017-09-06T10:38:12-04:00 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_errors=1 libbeat.es.publish.write_bytes=127
2017-09-06T10:38:42-04:00 INFO No non-zero metrics in the last 30s
2017-09-06T10:38:45-04:00 ERR Connecting error publishing events (retrying): Get http://loghost:5044: read tcp testmachine.ip.addr:37162->loghost.ip.addr:5044: read: connection reset by peer
2017-09-06T10:39:12-04:00 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_errors=1 libbeat.es.publish.write_bytes=127

[rdesanno (Rob)]

As a test, I'm reloading with http.cors disabled to see what happens but this is how I have my /etc/elasticsearch/elasticsearch.yml configured pre cors change:

cluster.name: elk01
node.name: ${HOSTNAME}
node.attr.rack: c3r207
http.cors.enabled: true
http.cors.allow-origin: "*"

thread_pool.index.size: 16
thread_pool.index.queue_size: 200

thread_pool.search.size: 25
thread_pool.search.queue_size: 1000

thread_pool.get.size: 16
thread_pool.get.queue_size: 1000

thread_pool.bulk.size: 16
thread_pool.bulk.queue_size: 75

indices.memory.index_buffer_size: 30%
indices.memory.min_index_buffer_size: 48mb

indices.fielddata.cache.size: 15%
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true

network.host: 0.0.0.0

[rdesanno (Rob)]

I think I see my problem. This is my filebeat configuration:

output.elasticsearch:
  hosts: ["loghost:5044"]

and believe it should be this:

output.logstash:
  hosts: ["loghost:5044"]

Testing now and so far seems to be my issue.

[rdesanno (Rob)]

Looks like that was the issue. Thanks to everyone for helping me work through this!