Problem connecting logstash and elasticsearch

[Belbo_belbo]

Hi, I'm doing a simple setup on minikube to practice with the stack. I have a sidecar pod with a container that generates logs and filebeats, then I have elasticsearch, logstash and kibana pods. I have this problem between the logstash and elasticsearch pods:

[2023-11-13T09:54:25,056][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
[2023-11-13T09:54:26,958][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"elasticsearch: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch: Name or service not known>}
[2023-11-13T09:54:26,959][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}

This is my logstash configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  labels:
    component: logstash
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: logstash
  template:
    metadata:
      labels:
        component: logstash
    spec:
      nodeSelector:
        kubernetes.io/hostname: minikube
      hostNetwork: true
      containers:
        - name: logstash
          image: logstash:8.11.0
          ports:
            - containerPort: 5044
          resources: {}
          volumeMounts:
            - name: logstash-config
              mountPath: /usr/share/logstash/pipeline
      volumes:
        - name: logstash-config
          configMap:
            name: logstash
---
apiVersion: v1
kind: Service
metadata:
  name: logstash
  labels:
    component: logstash
spec:
  ports:
  - port: 5044
  selector:
    component: logstash
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash
  labels:
    component: logstash
data:
  access-log.conf: |
    input {
      beats {
        port => "5044"
      }
    }
    output {
      elasticsearch {
        hosts => ["elasticsearch:9200"]
      }
    }     

This is my elasticsearch configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  labels:
    component: elasticsearch
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: elasticsearch
  template:
    metadata:
      labels:
        component: elasticsearch
    spec:
      nodeSelector:
        kubernetes.io/hostname: minikube
      hostNetwork: true
      containers:
        - name: elasticsearch
          image: elasticsearch:8.11.0
          ports:
            - containerPort: 9200
              name: client
          env:
            - name: discovery.type
              value: single-node
            - name: network.host
              value: 0.0.0.0
          resources: {}
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    component: elasticsearch
spec:
  ports:
  - port: 9200
    name: client
  selector:
    component: elasticsearch

[carly.richmond]

Hi @Belbo_belbo,

Welcome! Do you have a logstash pipeline at all? Could you share that configuration file as well?

[Belbo_belbo]

Isn't the pipeline the one inside the configmap? If yes, I attached it in the first file, it is located below

[carly.richmond]

Yes it is your right. Thanks for pointing that out!

Which version of Elasticsearch are you using? It looks like Logstash isn't able to access Elasticsearch so the connection might be being blocked somehow.

Have you setup the basic security as per the guide?

[Belbo_belbo]

This is my entire elasticsearch and logstash setup. The only safety things I've used are the ones you see

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-config
  namespace: logging
data:
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }


    output {
      elasticsearch {
        hosts => [ "https://elasticsearch:9200" ]
        ssl_certificate_verification => false
        user => "elastic"
        password => "mypass"
      }
    }

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  namespace: logging
spec:
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
        - name: logstash
          image: docker.elastic.co/logstash/logstash:8.11.0
          ports:
            - containerPort: 5044
          volumeMounts:
            - name: config-volume
              mountPath: /usr/share/logstash/pipeline/logstash.conf
              subPath: logstash.conf
      volumes:
        - name: config-volume
          configMap:
            name: logstash-config

---
apiVersion: v1
kind: Service
metadata:
  name: logstash
  namespace: logging
spec:
  ports:
    - port: 5044
      protocol: TCP
      targetPort: 5044
  selector:
    app: logstash
---
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: logging
spec:
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
        ports:
        - containerPort: 9200      

---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: logging
spec:
  ports:
    - port: 9200
      protocol: TCP
      targetPort: 9200
  selector:
    app: elasticsearch