Failed to publish long multiline events

[glipinski (Grzegorz Lipiński)]

Hi,

I am trying to send logs from Filebeat to Logstash

Filebeat 5.5.2 (Centos 6.9)
Logstash 5.5.2 (Windows Server 2008)

But I am getting an error:

INFO Error publishing events (retrying): read tcp ****** ->*****:5044: wsarecv: An established connection was aborted by the software in your host machine.

Error occures only for long events (i.e. 30000 characters) for short events logs are successfuly send to logstash -> elasticsearch and are visible in kibana.

Filebeat configuration:

filebeat.prospectors:


- input_type: log
  paths:
    - D:\Logs\*
  multiline.pattern: '(--+)'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 10000
  max_bytes: 10000000
  ignore_older: 24h

output.logstash:
  hosts: ["IP.ADD.RE.SS:5044"]

logging.level: debug

Logstash Pipeline:

input {
  beats {
    port => 5044
    client_inactivity_timeout => 60
  }
}

output {
  elasticsearch {
    hosts => ["IP.AD.DR.ESS:9200"]
    index => "test-%{+YYYY.MM.dd}"
    user => someuser
    password => somepassword
  }
}

[steffens (Steffen Siering)]

Hmm... That's interesting. I wonder if it's logstash or windows closing the connection. What happens if you set bulk_max_size: 1. This ensures at most one event being send to Logstash at a time (just for testing purposes). Also try to increase client_inactivity_timeout: 7200. Maybe it is the timer in Logstash only being reset after having parsed an event.

[glipinski (Grzegorz Lipiński)]

Hi,

Thanks for response.
Logstash and Filebeat configuration weren't an issue.
Traffic was terminated on firewall
Now all events are successfully send to Logstash.
I am not using max_bytes, client_inactivity_timeout anymore