Problems getting filebeat to connect to logstash container running in kubernetes

[RyanCherry]

Hi there, I have used the helm chart to install logstash onto my kubernetes cluster, however I am having problems getting filebeat to send logs to logstash.

Filebeat is running on a windows server as a service on that server, and previously had no issues sending to logstash running on a linux box. I need to move logstash onto kubernetes but filebeat logs keep showing the following:

2021-07-05T13:36:00.048+1200	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(async(tcp://route-to-container:443))
2021-07-05T13:36:00.048+1200	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-07-05T13:36:00.048+1200	INFO	[publisher]	pipeline/retry.go:223	  done
2021-07-05T13:36:00.049+1200	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(async(tcp://route-to-container:443)) established
2021-07-05T13:36:00.074+1200	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: EOF
2021-07-05T13:36:00.074+1200	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-07-05T13:36:00.075+1200	INFO	[publisher]	pipeline/retry.go:223	  done
2021-07-05T13:36:00.096+1200	ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: client is not connected
2021-07-05T13:36:00.096+1200	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-07-05T13:36:00.097+1200	INFO	[publisher]	pipeline/retry.go:223	  done

filebeat.yml

output.logstash:
  hosts: ["route-to-container:443"]
  loadbalance: true
  ssl.enabled: false

logstash.yml

    http.host: "0.0.0.0"
    log.level: info

logstash.conf (pipeline)

input {
      beats {
        port => 5001
        #client_inactivity_timeout => 3600 # 1 hour
      }
    }
output {
  stdout {
    codec => json
  }
}

Logstash does not show any errors in the container logs, or any logs that would suggest something is failing to connect
Connections are not being blocked by any firewalls
Kubernetes service is configured with

  ports:
    - name: beats
      port: 5001
      protocol: TCP
      targetPort: 5001

route-to-container targets the beats port, and is secured via Edge

I have not been able to find any information on how to fix this so far, any help would be greatly appreciated

[mtojek (Marcin Tojek)]

That's weird indeed. Did you check connectivity from inside of the pod? curl? openssl client?

[RyanCherry]

I am able to telnet from the windows server to the route I have defined in kubernetes, using port 443. From inside the logstash pod I can also curl and get a good response from the windows server.

Connectivity seems to be there for both of them

[RyanCherry]

I have found that I will be unable to implement this in the current state as our logstash deployment cannot receive TCP connections, only HTTP. Currently it seems there is no way to send from Filebeat to Logstash over HTTP, only TCP. Is this possible to get implemented? Are there any work arounds to this that can be recommended?