ERR SSL client failed to connect with: read tcp ->:5044: i/o timeout in kubernetes

Elastic Stack Beats
1

Hi,
I have created a one filebeat docker image. which i am using in kubernetes cluster. I am getting the below error.
transport.go:125: ERR SSL client failed to connect with: dial tcp 172.35.5.125:5044: getsockopt: connection refused
ERR SSL client failed to connect with: read tcp 100.96.8.7:38760->172.35.5.125:5044: i/o timeout.

I tried the below solutions

curl -v --cacert ssl.crt logstash.local-my.domain:5044

<acert ssl.crt logstash.local-my.domain:5044

  • Rebuilt URL to: logstash.local-my.domain:5044/
  • Trying 172.35.5.125...
  • Connected to 172.35.5.125 port 5044 (#0)

GET / HTTP/1.1
Host: :172.35.5.1255044
User-Agent: curl/7.47.0
Accept: /
But it isn't working. Also i am getting the below error.

file.go:77: WARN Overwriting type key

below is my filebeat.yml

filebeat:
prospectors: []
registry_file: "/var/lib/filebeat/registry"
config_dir: "/etc/filebeat/conf.d"
output:
logstash:
enabled: true
hosts:
- logstash.local-my.domain:5044
- logstash2.local-my.domain:5044
loadbalance: true
tls:
enabled: true
certificate: "/etc/ssl/certs/"
certificate_authorities:
- "/etc/ssl/certs/ssl.crt"
certificate_key: "/etc/ssl/private/ssl.key"
verification_mode: none
file:
enabled: false
path: "/tmp/filebeat"
filename: filebeat
rotate_every_kb: 1000
number_of_files: 7Preformatted text
logging:
level: info

# enable file rotation with default configuration
to_files: true

# do not log to syslog
to_syslog: false

files:
path: /var/log/mybeat
name: mybeat.log
keepfiles: 7

below is my kubernetes yml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: log
labels:
app: filebeat
spec:
template:
metadata:
labels:
app: filebeat
name: filebeat
spec:
imagePullSecrets:
- name: key
containers:
- name: filebeat
image: filebeat:v25
resources:
limits:
cpu: 50m
memory: 500Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: containers
mountPath: /var/log/containers
- name: dockerlib
mountPath: /var/lib/docker/containers
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: containers
hostPath:
path: /var/log/containers
- name: dockerlib
hostPath:
path: /var/lib/docker/containers
2

Looks like an SSL issue. Note that the protocol between Beats and LS is not HTTP, but a custom binary protocol, I recommend troubleshooting with something like: openssl s_client -connect logstash.local-my.domain:5044 -showcerts -tls1 -CAfile /etc/ssl/certs/ssl.crt

But I think the CAfile is supposed to be a "pem" file, not crt. Also, you have folder under certificate, it should be a file if you want, but you don't actually need it unless you are doing client authentication as well.

I recommend reviewing the settings here: https://www.elastic.co/guide/en/beats/filebeat/5.4/configuration-output-ssl.html

3

Hi @tudor

I am getting the below result

CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1494910790
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
4

@tudor I have removed the ssl config from the filebeat.yml

so now i am getting the following error

balance.go:248: INFO Error publishing events (retrying): EOF

My filebeat looks like below

filebeat:
  prospectors: []
  registry_file: "/var/lib/filebeat/registry"
  config_dir: "/etc/filebeat/conf.d"
output:
  logstash:
    enabled: true
    hosts:
    - logstash.local-my.domain:5044
    - logstash2.local-my.domain:5044
    loadbalance: true
  file:
    enabled: false
    path: "/tmp/filebeat"
    filename: filebeat
    rotate_every_kb: 1000
    number_of_files: 7
logging:
  level: info

  # enable file rotation with default configuration
  to_files: true

  # do not log to syslog
  to_syslog: false

  files:
    path: /var/log/mybeat
    name: mybeat.log
    keepfiles: 7
5

did you also disable SSL in logstash?

6

What @steffens wrote, that looks like SSL is still enabled on the LS side, you need to disable it on both if you want that.

7

I have changed the tag TLS to SSl in my filebeat.yml. So the ssl issue is solved.
Now i am facing another issue.

balance.go:248: INFO Error publishing events (retrying): EOF

8

There are quite a few posts in the forum with a similar error message. Please check if one of these resolves your issue.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.