ISPHOST
September 5, 2017, 1:29pm
1
Hello.
Error in the filebeat container:
2017/09/05 13:10:36.973731 tls.go:200: WARN SSL/TLS verifications disabled.
2017/09/05 13:10:41.974455 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/05 13:10:56.975332 sync.go:85: ERR Failed to publish events caused by: EOF
When I disable SSL in Logstash and Filebeat this error message disappears. But I can't do it on my prod env and looking for help to fix it.
Env info: #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
The configuration file of filebeat:
filebeat:
idle_timeout: 5s
prospectors:
- ignore_older: 10s
close_inactive: 5m
paths:
- /var/log/service.log
fields: {---SKIP---}
- ignore_older: 10s
close_inactive: 5m
paths:
- /var/log/myaudit.log
fields: {---SKIP---}
path:
data: /var/lib/beat
logs: /var/log/beat
output:
logstash:
enabled: true
hosts:
- 127.0.0.1:5043
ssl:
verification_mode: none
certificate: /usr/local/filebeat.crt
key: /usr/local/filebeat.key
certificate_authorities:
- /usr/local/filebeat.crt
timeout: 15
logging:
level: warning
PS: If needed I can add more information about env and config files.
steffens
September 5, 2017, 3:47pm
2
Use the </> button in the editors toolbar to properly format configuration and log files.
Do you have ssl, client authentication and certificates properly enabled in logstash? Can you share your logstash input configuration?
Some TLS/SSL implementations just close the connection if some validation in the TLS handshake fails. If logstash is not configured with SSL enabled, it will close the connection due to a parsing error.
ISPHOST
September 6, 2017, 7:30am
3
Part of logstash configuration file:
input {
beats {
port => 5043
ssl => true
ssl_certificate => "/usr/local/logstash/filebeat.crt"
ssl_key => "/usr/local/logstash/filebeat.key"
}
tcp {
port => 5042
codec => "json_lines"
}
}
steffens
September 6, 2017, 11:44am
4
So no client authentication. you don't need ssl.certificate and ssl.key in filebeat.
Setting up and troubleshooting SSL is quite a pain. Please check these docs for some instructions and tips on troubleshooting.
ISPHOST
September 6, 2017, 12:01pm
5
With disabled SSL in filebeat configuration same error.
2017/09/06 11:58:57.266448 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/06 11:58:57.266481 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/06 11:58:58.281227 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/06 11:58:58.281252 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/06 11:59:00.296434 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/06 11:59:00.296461 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/06 11:59:02.241690 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=3 libbeat.logstash.publish.read_errors=3 libbeat.logstash.publish.write_bytes=1934 libbeat.logstash.published_but_not_acked_events=18 libbeat.publisher.published_events=6 publish.events=2 registrar.states.current=2 registrar.states.update=2 registrar.writes=1
2017/09/06 11:59:02.243510 prospector_log.go:324: INFO File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /var/log/syslog.log
2017/09/06 11:59:04.311820 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/06 11:59:04.311842 single.go:91: INFO Error publishing events (retrying): EOF
ISPHOST
September 7, 2017, 8:23am
6
Could you please tell why do you think that is SSL client authentication disabled, if in logstash configuration option ssl is true?
steffens
September 7, 2017, 11:18am
7
Could you please tell why do you think that is SSL client authentication disabled, if in logstash configuration option ssl is true?
SSL/TLS is by default server authentication only. That is the client validates the server certificate, but the server applies no additional validation.
With Client authentication the client must also possess a signed certificate. The server will ask for the clients certificate and validate it. Client authentication must be explicitly enabled and configured.
In your filebeat config you even disable the certificate verification. That is, you disabled all authentication. Still, with these settings the traffic would be encrypted.
With disabled SSL in filebeat configuration same error.
This indicates it is no SSL/TLS problem. Seems like logstash is closing the connection. Is logstash running on same host? Which logstash version are you using?-d 'logstash') would be helpful, to see when the EOF occurs.
ISPHOST
September 11, 2017, 11:42am
8
Filebeat output with -d 'logstash' arg:
2017/09/11 11:40:14.549498 sync.go:53: DBG connect
2017/09/11 11:40:14.598344 sync.go:107: DBG Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:14.600219 sync.go:78: DBG 1 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:22.216252 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=4 libbeat.logstash.publish.read_bytes=2872 libbeat.logstash.publish.write_bytes=2221 libbeat.logstash.publish.write_errors=2 libbeat.logstash.published_and_acked_events=2 libbeat.logstash.published_but_not_acked_events=2 libbeat.publisher.published_events=2 publish.events=2 registrar.states.update=2 registrar.writes=2
2017/09/11 11:40:28.548723 sync.go:107: DBG Try to publish 2 events to logstash with window size 1
2017/09/11 11:40:28.551029 sync.go:78: DBG 0 events out of 2 events sent to logstash. Continue sending
2017/09/11 11:40:28.551061 sync.go:58: DBG close connection
2017/09/11 11:40:28.551083 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/11 11:40:28.551100 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/11 11:40:28.551115 sync.go:58: DBG close connection
2017/09/11 11:40:29.552701 sync.go:53: DBG connect
2017/09/11 11:40:29.581588 sync.go:107: DBG Try to publish 2 events to logstash with window size 1
2017/09/11 11:40:29.603494 sync.go:78: DBG 1 events out of 2 events sent to logstash. Continue sending
2017/09/11 11:40:29.603548 sync.go:107: DBG Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:29.613783 sync.go:78: DBG 1 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:43.549197 sync.go:107: DBG Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:43.549644 sync.go:78: DBG 0 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:43.549709 sync.go:58: DBG close connection
2017/09/11 11:40:43.549726 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/11 11:40:43.549737 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/11 11:40:43.549744 sync.go:58: DBG close connection
2017/09/11 11:40:44.549998 sync.go:53: DBG connect
2017/09/11 11:40:44.599014 sync.go:107: DBG Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:44.620172 sync.go:78: DBG 1 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:52.216171 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=4 libbeat.logstash.publish.read_bytes=2907 libbeat.logstash.publish.write_bytes=2633 libbeat.logstash.publish.write_errors=2 libbeat.logstash.published_and_acked_events=3 libbeat.logstash.published_but_not_acked_events=3 libbeat.publisher.published_events=3 publish.events=3 registrar.states.update=3 registrar.writes=2
2017/09/11 11:40:53.551136 sync.go:107: DBG Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:53.551640 sync.go:78: DBG 0 events out of 1 events sent to logstash. Continue sending
2017/09/11 11:40:53.551698 sync.go:58: DBG close connection
2017/09/11 11:40:53.551726 sync.go:85: ERR Failed to publish events caused by: EOF
2017/09/11 11:40:53.551741 single.go:91: INFO Error publishing events (retrying): EOF
2017/09/11 11:40:53.551754 sync.go:58: DBG close connection
2017/09/11 11:40:54.551985 sync.go:53: DBG connect
2017/09/11 11:40:54.581760 sync.go:107: DBG Try to publish 1 events to logstash with window size 1
2017/09/11 11:40:54.588978 sync.go:78: DBG 1 events out of 1 events sent to logstash. Continue sending
steffens
September 11, 2017, 12:56pm
9
This looks like a network or Logstash problem. This time the EOFs (connection being closed by remote) do happen while filebeat is writing/sending a single event to Logstash.
Which logstash version and logstash-beats-input plugin version have you installed?
ISPHOST
September 11, 2017, 1:14pm
10
As I said before, I am using:
# logstash -V
logstash 2.4.1
# logstash-plugin list --verbose
logstash-input-beats (3.1.8)
Upgrade to current version of logstash didn't help me.
steffens
September 12, 2017, 12:30pm
11
Assuming you are using logstash 5.5 try to set client_inactivity_timeout to 2 hours.
What else is logstash doing? Some grok pattern or some output blocking logstash.
Also check logstash logs again.
Can you try beats->logstash with this very minimal logstash (version 5.5) config first?
input {
beats {
port => 5044
client_inatcivity_timeout => 7200 # 2 hours
}
}
output {
null {}
}
This ensures logstash is doing no work and logstash is only closing connections after 2 hours of inactivity (same limit as default TCP connection timeout on linux). If you still get an EOF with this configuration, the connection is not closed by logstash, but your OS/firewall rules/... . Also get a trace with tcpdump you can inspect with wireshark. Is the TCP connection properly closed (FIN package) or do you just see the RST flag being set (in the later TCP connection state is dropped by server).
If you don't get the EOF, it's some problem with your logstash setup. Try adding the filters with null output first. Still no EOF, add the output. Still no EOF, add the tcp input.
system
October 10, 2017, 12:30pm
12
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.