# Filebeat fails connection to Logstash on Kubernetes

**URL:** <https://discuss.elastic.co/t/filebeat-fails-connection-to-logstash-on-kubernetes/351822>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 25, 2024, 5:12pm UTC](https://discuss.elastic.co/t/filebeat-fails-connection-to-logstash-on-kubernetes/351822 "2024-01-25T17:12:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ziggiyzoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziggiyzoo/32/131122_2.png) [@Ziggiyzoo](https://discuss.elastic.co/u/Ziggiyzoo)\
**Post date:** [January 25, 2024, 5:12pm UTC](https://discuss.elastic.co/t/filebeat-fails-connection-to-logstash-on-kubernetes/351822/1 "2024-01-25T17:12:48Z")

</div>

Hi,

I'm trying to connect a Filebeats deployment to Logstash but I am having error getting Filebeat to connect.

`A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.`

I've pretty much exhausted all that I can think of when it comes to why this isn't working, so any ideas would be appreciated! I've included the relevant configs below.

Filebeats is on Server A  
Logstash is on a Kubernetes Cluster

In the filebeat pipeline I have tried:

```auto
output.logstash:
  # The Logstash hosts
  hosts: ["kubernetes_host:5044"]
  ssl.enabled: true 

```

and

```auto
  # The Logstash hosts
  hosts: ["kubernetes_host:443/logstash-ingress"]
  ssl.enabled: true 

```

And my logstash pipeline beats input looks like this:

```auto
input {
  beats {
      id => "beats-input"
      port => "5044"
      ssl => true
      ssl_certificate => "path_to_cert"
      ssl_key => "path_to_key"
      ssl_verify_mode => "none"
  }
}

```

I for the logstash deployment on Kubernetes I am deploying using Helm & Ansible Playbooks.

values.yml contains:

```auto
values:
          replicas: 1
          extraPorts:
          - name: beats
            containerPort: 5044
          clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s" # Elastic does not go into green status when using 1 replica
          envFrom:
          - secretRef:
              name: logstash_creds
          
          # Configure Readiness & Liveness Probe to 
          readinessProbe:
            httpGet:
              path: /
              port: 9600
            initialDelaySeconds: 180
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
            successThreshold: 3

          # Configure Beats Service
          service:
            annotations: {}
            type: ClusterIP
            loadBalancerIP: ""
            ports:
              - name: beats
                port: 5044
                protocol: TCP
                targetPort: 5044
          
          # Add TLS Certificate
          secretMounts:
            - name: ##
              secretName: ##
              path: path_to_certs

```

I have also added a ConfigMap for allowing a TCP connection with NGINX following [this guide](https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/exposing-tcp-udp-services.md), and a ingress for when I was trying to use /logstash-ingress path.

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [January 26, 2024, 3:07am UTC](https://discuss.elastic.co/t/filebeat-fails-connection-to-logstash-on-kubernetes/351822/2 "2024-01-26T03:07:31Z")

</div>

I think the first step would be to determine if logstash is actually accessible on Port 5044, can you use telnet or a TCP connection tester to see if you can actually connect to that port on the logstash instance?

---

<div class="post-metadata">

**Author:** ![Ziggiyzoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziggiyzoo/32/131122_2.png) [@Ziggiyzoo](https://discuss.elastic.co/u/Ziggiyzoo)\
**Post date:** [January 26, 2024, 12:12pm UTC](https://discuss.elastic.co/t/filebeat-fails-connection-to-logstash-on-kubernetes/351822/3 "2024-01-26T12:12:24Z")

</div>

Hi,

I've now tried using netcat to test the port from inside the cluster with an alpine deployment. I got the following results:

`nc -z "logstash-service" 5044` returned a successful connection

`nc -z "logstash-pod-ip" 5044 returned a successful connection

I've also tried telnet from my device:

`telnet "kubernetes-cluster"/logstash"` and `telnet "kubernetes-cluster":5044` both fail to connect.

I can, from any of the devices tried so far, connect to to the logstash api, port 9600, with `"kubernetes-cluster"/logstash-api` this succeeds.

I feel like this is more of an issue with me not understanding networking, than with filebeat or logstash 😃

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [January 26, 2024, 1:15pm UTC](https://discuss.elastic.co/t/filebeat-fails-connection-to-logstash-on-kubernetes/351822/4 "2024-01-26T13:15:19Z")

</div>

I'm not too familiar with the setup you've got but it sounds like the port you're using that uses the built-in ingress is working fine and the port that's going through nginx is not and so maybe looking at the nginx config is the next step?

---

<div class="post-metadata">

**Author:** ![Ziggiyzoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziggiyzoo/32/131122_2.png) [@Ziggiyzoo](https://discuss.elastic.co/u/Ziggiyzoo)\
**Post date:** [January 26, 2024, 1:33pm UTC](https://discuss.elastic.co/t/filebeat-fails-connection-to-logstash-on-kubernetes/351822/5 "2024-01-26T13:33:28Z")

</div>

When it comes to kubernetes ingress with nginx I am familiar with the use of DNS/IP & Path. But I remember reading in another forum post here/stack overflow, that filebeat does not accept "host/path" as an output, only "host:port"?

This is what's causing the biggest headache for me!

I've tried connecting our filebeat kubernetes deployment to logstash.. it has connected and is sending logs..

`output: "Logstash-Transfer-Service":5044`  
to  
Transfer Service (Namespace B) -\> Logstash Service (Namespace A)  
to  
Logstash Pod

This worked after I disabled SSL.

I think I have narrowed down the issues to:

SSL  
NGINX Ingress

I will update you if I find a solution!

---

<div class="post-metadata">

**Author:** ![Ziggiyzoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziggiyzoo/32/131122_2.png) [@Ziggiyzoo](https://discuss.elastic.co/u/Ziggiyzoo)\
**Post date:** [January 29, 2024, 2:14pm UTC](https://discuss.elastic.co/t/filebeat-fails-connection-to-logstash-on-kubernetes/351822/6 "2024-01-29T14:14:19Z")

</div>

I've created a new ingress on kubernetes! This has allowed filebeat to connect to logstash.

K8s Ingress:

```auto
type or paste code hereapiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: filebeat
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - "....devlogstash...."
      secretName: cluster-crt
  rules:
    - host: "....devlogstash...."
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: logstash-transfer
              port:
                number: 5044

```

This seems to work as it's the only ingress using this host so I don't need to use a path.. letting me use `"DNS:443"` in filebeat output.

This feels like more of a temporary fix, as it requires adding an additional DNS to the filebeat servers, when it would be preferable to use the original K8s DNS with a TCP ingress etc.

But I've not moved on to having an issue now with lumberjack protocol error:

```auto
2024-01-29T14:03:53.534Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(async(tcp://....devlogstash....:443))
2024-01-29T14:03:53.534Z	INFO	[publisher]	pipeline/retry.go:213	retryer: send wait signal to consumer
2024-01-29T14:03:53.534Z	INFO	[publisher]	pipeline/retry.go:217 done
2024-01-29T14:03:53.545Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(async(tcp://devlogstash.devs.facilities.rl.ac.uk:443)) established
2024-01-29T14:03:53.545Z	INFO	[publisher]	pipeline/retry.go:213	retryer: send wait signal to consumer
2024-01-29T14:03:53.545Z	INFO	[publisher]	pipeline/retry.go:217 done
2024-01-29T14:03:53.570Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2024-01-29T14:03:53.570Z	INFO	[publisher]	pipeline/retry.go:223 done
2024-01-29T14:03:53.572Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: lumberjack protocol error
2024-01-29T14:03:53.572Z	INFO	[publisher]	pipeline/retry.go:213	retryer: send wait signal to consumer
2024-01-29T14:03:53.572Z	INFO	[publisher]	pipeline/retry.go:217 done
2024-01-29T14:03:53.574Z	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: client is not connected

```

I'm sure other people have had that error, so I will go and look for solutions and hopefully be able to mark this solved 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 26, 2024, 4:14pm UTC](https://discuss.elastic.co/t/filebeat-fails-connection-to-logstash-on-kubernetes/351822/7 "2024-02-26T16:14:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.