Connecting Filebeat to and ECK from outside the K8s with ingress

[Carlos_T]

Good morning.

From a Linux server out of the K8s where my ES is running, Im trying to setup the filebeat so it connects to the Service using the ingress.

If from the linux where I run the filebeat I run a curl to check the connectivity , it works:

Now Im trying to set up the filebeat.yaml with this settings:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elasticsearcheck.XXXXXXXXXXXXXXXX.com"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "${FB_USER}"
  password: "${FB_PWD}"
  ssl.verification_mode: "none"

Once I run the 'filebeat setup -e' I get the following error.

{"log.level":"error","@timestamp":"2023-06-08T12:54:58.624+0200","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":235},"message":"error connecting to Elasticsearch at https://elasticsearcheck.XXXXXXX..com:9200: Get "https://elasticsearcheck.XXXXXXX.com:9200": dial tcp X.X.X.X:9200: connect: no route to host","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-06-08T12:54:58.624+0200","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://elasticsearcheck.XXXXXXX.com:9200: Get "https://elasticsearcheck.XXXXXXX.com:9200": dial tcp X.X.X.X:9200: connect: no route to host]","service.name":"filebeat","ecs.version":"1.6.0"}

Said that. As my curl really resolves an IP:9200 without addind manually the 9200 as it is included in the ingress. I've got the feeling that the error Filebeat is having is about translating the output.elasticsearch.hosts to -> IP:9200:9200

Does what Im saying makes any sense?

What do you think it could be the problem here? As I say I dont think there is a problem of firewalls or passwords as the curl works and the error message talks about what IMHO is a wrong host URL PATH.

Thank you in advance

[ccaillet (Christophe Caillet)]

Hi all,

I think that it depends of your CNI on kubernetes. In fact if you masquerade the output from your cluster to the external you must ensure that a worker could join the ES instance if not it explains why you have "no route to host". The pod try to connect to ES instance and use the "routing" method used internally on your cluster. Here I use cilium as CN and permit the usage of the local routing table of the nodes for routing trafic intern (pod) -> extern, the CNI cilium masquerade from each node traffic.

I think it's just that. If the node cannot join the ES instance you could maybe define an http proxy (which is joinable from the nodes) and define the right configuration to permit filebeat to go outside through the proxy.

Regards

[Carlos_T]

Thank you ccaillet.

I've finally fixed it by adding this line to my ingress.yaml

nginx.ingress.kubernetes.io/proxy-body-size: "0"

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kibana
  namespace: monitoring
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
spec:
  ingressClassName: nginx
  rules:
    - host: kibanaeck.XXXXXXXXXcom
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kibana-kb-http
                port:
                  name: https

Thank you very much again