ERR Failed to publish events caused by: EOF (no tls)

[lolo67]

Hello,

In filebeat.log I have many :
ERR Failed to publish events caused by: EOF

I checked that my filebeat.yml is valid :
http://www.yamllint.com/

I checked that ssl is used and not tls.
I checked the logstash log : no error

It's filebeat 5.0.2.
I launched it in debug mode :`

/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -e -d '*'
... ... ...
 
2016/12/08 16:52:04.449724 prospector_log.go:79: DBG  Prospector states cleaned up. Before: 26, After: 26
2016/12/08 16:52:09.451856 log_file.go:84: DBG  End of file reached: /opt/WSO2/wso2esb/repository/logs/http_access_management_console_2016-12-01.log; Backoff now.
2016/12/08 16:52:10.863401 log_file.go:84: DBG  End of file reached: /opt/WSO2/wso2esb/repository/logs/wso2carbon.log; Backoff now.
2016/12/08 16:52:13.418002 client.go:194: DBG  handle error: EOF
2016/12/08 16:52:13.418067 client.go:110: DBG  closing
2016/12/08 16:52:13.418235 sync.go:78: DBG  0 events out of 1792 events sent to logstash. Continue sending
2016/12/08 16:52:13.418259 sync.go:58: DBG  close connection
2016/12/08 16:52:13.418274 sync.go:85: ERR Failed to publish events caused by: EOF
2016/12/08 16:52:13.418287 single.go:91: INFO Error publishing events (retrying): EOF
2016/12/08 16:52:13.418297 sync.go:58: DBG  close connection
2016/12/08 16:52:13.418419 single.go:156: DBG  send fail
2016/12/08 16:52:14.418605 sync.go:53: DBG  connect
2016/12/08 16:52:14.432454 sync.go:107: DBG  Try to publish 1792 events to logstash with window size 958
2016/12/08 16:52:14.444259 prospector.go:137: DBG  Run prospector
2016/12/08 16:52:14.444291 prospector_log.go:71: DBG  Start next scan

Any idea ?

Laurent

[ruflin]

There were quite a few previous threads with similar questions. Perhaps one of these could help? https://discuss.elastic.co/search?q=ERR%20Failed%20to%20publish%20events%20caused%20by%3A%20EOF If not, please provide the filebeat and logstash config file.

[lolo67]

Thanks Ruflin, indeed I did look at the other threads on the same subject, and ckecked for deprecated tls and ill-formated yml. But still, I couldn't resolve the problem.
So here's my filebeat.yml :

filebeat:
  prospectors:
    -
      paths:
        - /opt/WSO2/wso2esb/repository/logs/wso2carbon.log
      document_type: log
      fields:
         origin: SECURI_ESB
         category: TECHNIQUE
      ignore_older: 744h
    -
      paths:
        - /opt/WSO2/wso2esb/repository/logs/http_access_*.log
      document_type: log
      fields:
         origin: SECURI_ESB
         category: ACCESS
      ignore_older: 744h
    -
      paths:
        - /opt/WSO2/wso2is/repository/logs/wso2carbon.log
      document_type: log
      fields:
         origin: ESBMET_IS
         category: METIER
      ignore_older: 744h
    -
      paths:
        - /opt/WSO2/wso2is/repository/logs/http_access_*.log
      document_type: log
      fields:
         origin: ESBMET_IS
         category: ACCESS
      ignore_older: 744h
    -
      paths:
        - /opt/WSO2/wso2greg/repository/logs/wso2carbon.log
      document_type: log
      fields:
         origin: SECURI_REGISTRY
         category: METIER
      ignore_older: 744h
    -
      paths:
        - /opt/WSO2/wso2greg/repository/logs/http_access_*.log
      document_type: log
      fields:
         origin: SECURI_REGISTRY
         category: ACCESS
      ignore_older: 744h
    -
      paths:
        - /data/logs/*.log
      document_type: log
      fields:
         origin: ESBTEC_MYSQL
         category: TECHNIQUE
      ignore_older: 744h
    -
      paths:
        - /var/log/rabbitmq/*.log
      document_type: log
      fields:
         origin: HELIOS_RABBITMQ
         category: TECHNIQUE
      ignore_older: 744h
output:
  logstash:
    hosts: ["elk-docker:5044"]
    ssl:
      #insecure: true
      certificate_authorities: ["/etc/pki/tls/certs/logstash-beats.crt"]
logging:
  level: warning
  to_files: true
  to_syslog: false

  files:
    path: /var/log/filebeat
    name: filebeat.log
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7

And for the logstash part, 02-beats-input.conf and 30-output.conf :

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
    ssl_key => "/etc/pki/tls/private/logstash-beats.key"
  }
}

output {
  elasticsearch {
    hosts => ["localhost"]
    manage_template => false
    index => "filebeat-%{+YYYY.MM.dd.HH}"
    document_type => "%{[@metadata][type]}"
  }
}

Regards
Laurent

[steffens (Steffen Siering)]

which logstash version are you using? Please ensure you're using the most recent logstash-input-beats plugin.

Is it working with insecure: true?

[lolo67]

As shown in previous reply with logstash conf files, 02-beats-input.conf and 30-output.conf :
No it's not it's "insecure: true", it's "ssl => true"

/opt/logstash/bin/logstash --version
logstash 5.0.2

[steffens (Steffen Siering)]

I've been asking if events are published if insecure: true is enabled in filebeat or if you still get EOF. EOF means, the connection has been closed by remote host (saying logstash).

Also update logstash or at least the beats input plugin to most recent version.