"There are no ingest nodes in this cluster, unable to forward request to an ingest node" errors

Elastic Observability APM
,
1

Kibana version: 8.7.1

Elasticsearch version: 8.7.1

APM Server version: 8.7.1

APM Agent language and version: Node.JS 19

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: my-apmserver
  labels:
    app: apmserver
    component: eck
spec:
  version: 8.7.1
  count: 3
  secureSettings:
    - secretName: ny-es-elastic-user
  config:
    output:
      elasticsearch:
        hosts: ["my-es-internal-http.default:9200"]
        username: elastic
        password: "${elastic}"
        protocol: "https"
        ssl.certificate_authorities:
          ["/usr/share/apm-server/config/elasticsearch-ca/tls.crt"]
  podTemplate:
    spec:
      containers:
        - name: apm-server
          volumeMounts:
            - mountPath: /usr/share/apm-server/config/elasticsearch-ca
              name: elasticsearch-ca
              readOnly: true
      volumes:
        - name: elasticsearch-ca
          secret:
            defaultMode: 420
            optional: false
            secretName: elasticsearch-eck-ca # This is the secret that holds the Elasticsearch CA cert

[my-apm-server-985bdf477-hkpfv apm-server] {"log.level":"error","@timestamp":"2023-05-06T08:44:10.367Z","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":256},"message":"bulk indexing request failed","service.name":"apm-server","error":{"message":"flush failed: [500 Internal Server Error] {\"error\":{\"root_cause\":[{\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"}],\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"},\"status\":500}"},"ecs.version":"1.6.0"}

What should I do?

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

2

@Kok_How_Teh are you also using ECK to set up Elasticsearch? The error suggests that none of your Elasticsearch nodes have the "ingest" role. See Node configuration | Elastic Cloud on Kubernetes [2.7] | Elastic for how to set that up.

3

Resolved. Thanks.

1 Like
4

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.