Sync.go:85: ERR Failed to publish events caused by: EOF

ISPHOST · September 5, 2017, 1:29pm

Hello.
I am using Elasticsearch, Logstash, Kibana Docker images and have some troubles.

Error in the filebeat container:

2017/09/05 13:10:36.973731 tls.go:200: WARN SSL/TLS verifications disabled.
2017/09/05 13:10:41.974455 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/05 13:10:56.975332 sync.go:85: ERR Failed to publish events caused by: EOF

When I disable SSL in Logstash and Filebeat this error message disappears. But I can't do it on my prod env and looking for help to fix it.

Env info:
CentOS Linux release 7.3.1611 (Core)
Linux 1b08b588dbb9 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
filebeat version 5.3.2 (amd64), libbeat 5.3.2

The configuration file of filebeat:

filebeat:
  idle_timeout: 5s
  prospectors:
    - ignore_older: 10s
      close_inactive: 5m
      paths:
        - /var/log/service.log
      fields: {---SKIP---}
    - ignore_older: 10s
      close_inactive: 5m
      paths:
        - /var/log/myaudit.log
      fields: {---SKIP---}
 path:
   data: /var/lib/beat
   logs: /var/log/beat
 
 output:
   logstash:
     enabled: true
     hosts:
       - 127.0.0.1:5043
     ssl:
       verification_mode: none
       certificate: /usr/local/filebeat.crt
       key: /usr/local/filebeat.key
       certificate_authorities:
         - /usr/local/filebeat.crt
     timeout: 15
 
 logging:
   level: warning

PS: If needed I can add more information about env and config files.

steffens · September 5, 2017, 3:47pm

Use the </> button in the editors toolbar to properly format configuration and log files.

Do you have ssl, client authentication and certificates properly enabled in logstash? Can you share your logstash input configuration?

Some TLS/SSL implementations just close the connection if some validation in the TLS handshake fails. If logstash is not configured with SSL enabled, it will close the connection due to a parsing error.

ISPHOST · September 6, 2017, 7:30am

Part of logstash configuration file:

input {
        beats {
          port => 5043
      ssl => true
          ssl_certificate => "/usr/local/logstash/filebeat.crt"
          ssl_key => "/usr/local/logstash/filebeat.key"
        }
        tcp {
                port => 5042
                codec => "json_lines"
        }
}

steffens · September 6, 2017, 11:44am

So no client authentication. you don't need ssl.certificate and ssl.key in filebeat.

Setting up and troubleshooting SSL is quite a pain. Please check these docs for some instructions and tips on troubleshooting.

ISPHOST · September 6, 2017, 12:01pm

With disabled SSL in filebeat configuration same error.

2017/09/06 11:58:57.266448 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/06 11:58:57.266481 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/06 11:58:58.281227 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/06 11:58:58.281252 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/06 11:59:00.296434 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/06 11:59:00.296461 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/06 11:59:02.241690 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=3 libbeat.logstash.publish.read_errors=3 libbeat.logstash.publish.write_bytes=1934 libbeat.logstash.published_but_not_acked_events=18 libbeat.publisher.published_events=6 publish.events=2 registrar.states.current=2 registrar.states.update=2 registrar.writes=1
2017/09/06 11:59:02.243510 prospector_log.go:324: INFO File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /var/log/syslog.log
2017/09/06 11:59:04.311820 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/06 11:59:04.311842 single.go:91: INFO Error publishing events (retrying): EOF

ISPHOST · September 7, 2017, 8:23am

Could you please tell why do you think that is SSL client authentication disabled, if in logstash configuration option ssl is true?

steffens · September 7, 2017, 11:18am

Could you please tell why do you think that is SSL client authentication disabled, if in logstash configuration option ssl is true?

SSL/TLS is by default server authentication only. That is the client validates the server certificate, but the server applies no additional validation.

With Client authentication the client must also possess a signed certificate. The server will ask for the clients certificate and validate it. Client authentication must be explicitly enabled and configured.

In your filebeat config you even disable the certificate verification. That is, you disabled all authentication. Still, with these settings the traffic would be encrypted.

With disabled SSL in filebeat configuration same error.

This indicates it is no SSL/TLS problem. Seems like logstash is closing the connection. Is logstash running on same host? Which logstash version are you using?
Anything in logstash or beats logs? Some filebeat debug log (run with -d 'logstash') would be helpful, to see when the EOF occurs.

ISPHOST · September 11, 2017, 11:42am

Filebeat output with -d 'logstash' arg:

2017/09/11 11:40:14.549498 sync.go:53: DBG  connect
2017/09/11 11:40:14.598344 sync.go:107: DBG  Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:14.600219 sync.go:78: DBG  1 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:22.216252 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=4 libbeat.logstash.publish.read_bytes=2872 libbeat.logstash.publish.write_bytes=2221 libbeat.logstash.publish.write_errors=2 libbeat.logstash.published_and_acked_events=2 libbeat.logstash.published_but_not_acked_events=2 libbeat.publisher.published_events=2 publish.events=2 registrar.states.update=2 registrar.writes=2
2017/09/11 11:40:28.548723 sync.go:107: DBG  Try to publish 2 events to logstash with window size 1
2017/09/11 11:40:28.551029 sync.go:78: DBG  0 events out of 2 events sent to logstash. Continue sending
2017/09/11 11:40:28.551061 sync.go:58: DBG  close connection
2017/09/11 11:40:28.551083 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/11 11:40:28.551100 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/11 11:40:28.551115 sync.go:58: DBG  close connection
2017/09/11 11:40:29.552701 sync.go:53: DBG  connect
2017/09/11 11:40:29.581588 sync.go:107: DBG  Try to publish 2 events to logstash with window size 1
2017/09/11 11:40:29.603494 sync.go:78: DBG  1 events out of 2 events sent to logstash. Continue sending
2017/09/11 11:40:29.603548 sync.go:107: DBG  Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:29.613783 sync.go:78: DBG  1 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:43.549197 sync.go:107: DBG  Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:43.549644 sync.go:78: DBG  0 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:43.549709 sync.go:58: DBG  close connection
2017/09/11 11:40:43.549726 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/11 11:40:43.549737 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/11 11:40:43.549744 sync.go:58: DBG  close connection
2017/09/11 11:40:44.549998 sync.go:53: DBG  connect
2017/09/11 11:40:44.599014 sync.go:107: DBG  Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:44.620172 sync.go:78: DBG  1 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:52.216171 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=4 libbeat.logstash.publish.read_bytes=2907 libbeat.logstash.publish.write_bytes=2633 libbeat.logstash.publish.write_errors=2 libbeat.logstash.published_and_acked_events=3 libbeat.logstash.published_but_not_acked_events=3 libbeat.publisher.published_events=3 publish.events=3 registrar.states.update=3 registrar.writes=2
2017/09/11 11:40:53.551136 sync.go:107: DBG  Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:53.551640 sync.go:78: DBG  0 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:53.551698 sync.go:58: DBG  close connection
2017/09/11 11:40:53.551726 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/11 11:40:53.551741 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/11 11:40:53.551754 sync.go:58: DBG  close connection
2017/09/11 11:40:54.551985 sync.go:53: DBG  connect
2017/09/11 11:40:54.581760 sync.go:107: DBG  Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:54.588978 sync.go:78: DBG  1 events out of 1 events sent to logstash. Continue sending

steffens · September 11, 2017, 12:56pm

This looks like a network or Logstash problem. This time the EOFs (connection being closed by remote) do happen while filebeat is writing/sending a single event to Logstash.

Which logstash version and logstash-beats-input plugin version have you installed?

ISPHOST · September 11, 2017, 1:14pm

As I said before, I am using:

# logstash -V
logstash 2.4.1

# logstash-plugin list --verbose
logstash-input-beats (3.1.8)

Upgrade to current version of logstash didn't help me.

steffens · September 12, 2017, 12:30pm

Assuming you are using logstash 5.5 try to set client_inactivity_timeout to 2 hours.

What else is logstash doing? Some grok pattern or some output blocking logstash.

Also check logstash logs again.

Can you try beats->logstash with this very minimal logstash (version 5.5) config first?

input {
  beats {
    port => 5044
    client_inatcivity_timeout => 7200 # 2 hours
  }
}

output {
  null {}
}

This ensures logstash is doing no work and logstash is only closing connections after 2 hours of inactivity (same limit as default TCP connection timeout on linux). If you still get an EOF with this configuration, the connection is not closed by logstash, but your OS/firewall rules/... . Also get a trace with tcpdump you can inspect with wireshark. Is the TCP connection properly closed (FIN package) or do you just see the RST flag being set (in the later TCP connection state is dropped by server).

If you don't get the EOF, it's some problem with your logstash setup. Try adding the filters with null output first. Still no EOF, add the output. Still no EOF, add the tcp input.

system · October 10, 2017, 12:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.