Winlogbeat Nginx Proxy to Logstash - Connection Reset


(Douglas Rae) #1

Hello,

I have the following setup:

[Nginx Port 80]

Forwards to

[Logstash Port 5044]

Directly connects and speaks with

[Elasticsearch Port 9200]

This works with Metricbeat fine, all requests and conversations happen.

Here is my nginx setup:

user root;
worker_processes auto;
load_module /usr/lib/nginx/modules/ngx_stream_module.so;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
  worker_connections 1024;
}

stream {
log_format    basic    '$time_iso8601 $remote_addr '
                   '$protocol $status $bytes_sent $bytes_received '
                   '$session_time $upstream_addr '
                   '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

# Enable access_log statements for debugging

access_log /var/log/nginx/stream.log basic;

upstream syslog_servers {
server logstash:514;
}

server {
listen 514;
listen 514 udp;

proxy_responses 0;
proxy_pass syslog_servers;
proxy_buffer_size 4096k;

# access_log /var/log/nginx/stream.log basic;
}

upstream logstash_beats_servers {
server logstash:5044;
}

server {
listen 80;
proxy_responses 1;

proxy_connect_timeout 30s;

proxy_pass logstash_beats_servers;
proxy_buffer_size 4096k;

access_log /var/log/nginx/winlogbeat.log basic;
}

}

daemon off;

Here is my beats configuration:

input {
  beats {
    port => 5044
    type => beats
  }
}

output {
  if [type] == "beats" {
    elasticsearch {
    hosts => "elasticsearch:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    }
  }
}

I have captured the error in Nginx:

2018/05/15 14:23:50 [error] 1794#1794: *527214 recv() failed (104: Connection reset by peer) while proxying connection, client: 192.168.1.10, server: 0.0.0.0:80, upstream: "172.18.0.10:5044", bytes from/to client:3876/18, bytes from/to upstream:18/3876
2018/05/15 14:28:01 [error] 1795#1795: *528980 recv() failed (104: Connection reset by peer) while proxying connection, client: 192.168.1.10, server: 0.0.0.0:80, upstream: "172.18.0.11:5044", bytes from/to client:1669/6, bytes from/to upstream:6/1669
2018/05/15 14:29:36 [error] 1795#1795: *529380 recv() failed (104: Connection reset by peer) while proxying connection, client: 192.168.1.10, server: 0.0.0.0:80, upstream: "172.18.0.12:5044", bytes from/to client:3196/12, bytes from/to upstream:12/3196

I have also captured the problem in Winlogbeat log:

2018-05-15T16:05:56.893+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: EOF
2018-05-15T16:05:56.895+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: client is not connected
2018-05-15T16:05:57.896+0100	ERROR	pipeline/output.go:92	Failed to publish events: client is not connected

I am unsure what the difference in the conversation is between Metricbeat -> Logstash and Winlogbeat -> Logstash, I was of the understanding they both used the lumberjack protocol on 5044


(Pier-Hugues Pellerin) #2

Winlogbeat and Metricbeat both uses the same exact code for the Lumberjack output. I haven't used Nginx as a proxy other than with the HTTP protocol. Since Lumberjack is not based on HTTP I am not sure how it will behave in that condition, since Lumberjack is a persistent connection with two way communication.

2018/05/15 14:23:50 [error] 1794#1794: *527214 recv() failed (104: Connection reset by peer) while proxying connection, client: 192.168.1.10, server: 0.0.0.0:80, upstream: "172.18.0.10:5044", bytes from/to client:3876/18, bytes from/to upstream:18/3876
2018/05/15 14:28:01 [error] 1795#1795: *528980 recv() failed (104: Connection reset by peer) while proxying connection, client: 192.168.1.10, server: 0.0.0.0:80, upstream: "172.18.0.11:5044", bytes from/to client:1669/6, bytes from/to upstream:6/1669
2018/05/15 14:29:36 [error] 1795#1795: *529380 recv() failed (104: Connection reset by peer) while proxying connection, client: 192.168.1.10, server: 0.0.0.0:80, upstream: "172.18.0.12:5044", bytes from/to client:3196/12, bytes from/to upstream:12/3196

This look like Logstash killed the connection? Maybe because it hit the client_timeout value on the beat input side?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.