# There are no ingest nodes in this cluster, unable to forward request to an ingest node

**URL:** <https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740>\
**Category:** APM\
**Tags:** server\
**Created:** [July 27, 2021, 2:27pm UTC](https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740 "2021-07-27T14:27:27Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![nitin194](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitin194/32/92246_2.png) [@nitin194](https://discuss.elastic.co/u/nitin194)\
**Post date:** [July 27, 2021, 2:27pm UTC](https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740/1 "2021-07-27T14:27:28Z")

</div>

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text 🙂

TIP 1: select at least one tag that further categorizes your topic. For example `server` for APM Server related questions, `java` for questions regarding the Elastic APM Java agent, or `ui` for questions about the APM App within Kibana.

TIP 2: Check out the [troubleshooting guide](https://www.elastic.co/guide/en/apm/get-started/current/troubleshooting.html) first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

**Kibana version** :  
7.13.4  
**Elasticsearch version** :  
7.13.4  
**APM Server version** :  
7.13.4  
**APM Agent language and version** :  
NA  
**Browser version** :  
NA  
**Original install method (e.g. download page, yum, deb, from source, etc.) and version**:  
Helm  
**Fresh install or upgraded from other version?**  
Fresh  
**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.  
Nothing special  
**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:  
We are using below setup all installed through helm charts

1. 3 Master pods
2. 2 Data pods
3. 3 remote client pods
4. 1 kibana pod
5. 1 apm-server pod  
**Steps to reproduce** :
6. Start APM-sever deployment
7. 
8. 

**Errors in browser console (if relevant)**:  
NA  
**Provide logs and/or server output (if relevant)**:

````auto
{"log.level":"info","@timestamp":"2021-07-27T13:52:24.351Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":213},"message":"retryer: send wait signal to consumer","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-07-27T13:52:24.351Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":217},"message":" done","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-07-27T13:52:24.353Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":224},"message":"failed to perform any bulk index operations: 500 Internal Server Error: {\"error\":{\"root_cause\":[{\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"}],\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"},\"status\":500}","ecs.version":"1.6.0"}```

Is it necessary to have a ingest node for apm-server to work?
is it advisable to allow ingest role in the same remote client node?
how to make apm-server work without ingest nodes if we are using external ingestion like logstash?

Regards
Nitin
````

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [July 28, 2021, 12:17am UTC](https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740/2 "2021-07-28T00:17:20Z")

</div>

> Is it necessary to have a ingest node for apm-server to work?

In its default configuration, yes. Per [Configure the Elasticsearch output | APM User Guide [8.11] | Elastic](https://www.elastic.co/guide/en/apm/server/current/elasticsearch-output.html#pipeline-option-es), you can disable the ingest pipeline by configuring apm-server with `output.elasticsearch.pipeline: _none`. Although it is possible, I wouldn't recommend it -- this might break the UI. We are making increasingly more use of ingest node for processing events as they are indexed.

> is it advisable to allow ingest role in the same remote client node?

I'm afraid I don't understand your question, but you can find information about Elasticsearch node roles at [Node | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html)

> how to make apm-server work without ingest nodes if we are using external ingestion like logstash?

If you're using the Logstash output, you can instruct Logstash to pass the pipeline onto Elasticsearch, as described in [Use ingest pipelines for parsing | Logstash Reference [7.13] | Elastic](https://www.elastic.co/guide/en/logstash/7.13/use-ingest-pipelines.html). You would need to load the ingest pipeline directly in Elasticsearch.

---

<div class="post-metadata">

**Author:** ![nitin194](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitin194/32/92246_2.png) [@nitin194](https://discuss.elastic.co/u/nitin194)\
**Post date:** [July 28, 2021, 5:43am UTC](https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740/3 "2021-07-28T05:43:49Z")

</div>

Hello @axw ,

Thank you for your response. we are completely new to implement APM server. The motive to use APM server is to collect and visualize application API level metrics via open telemetry collector(elastic exporter) pass it to ES via APM server and then visualize them through Kibana. But we don't know how to set this whole pipeline. As per your suggestion we have converted remote client node role into "coordinating node + ingest node" role. The previous error has vanished now. But we are not sure weather to use indices or pipeline to feed into Elasticsearch to achieve our purpose. Nor we could find any relevant documentation or article for such settings. Also, the APM page on Kibana shows no apm services installed.

 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/f/3/f38ea39ed508b056c4360e2e57efe5379bc304a1.png)

Could you please hint us so that we can use correct APM settings? We are very keen on utilizing APM for our use case

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [July 28, 2021, 6:49am UTC](https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740/4 "2021-07-28T06:49:45Z")

</div>

> The motive to use APM server is to collect and visualize application API level metrics via open telemetry collector(elastic exporter) pass it to ES via APM server and then visualize them through Kibana.

The `elastic` exporter in opentelemetry-collector-contrib is deprecated, and has been replaced by native support for OpenTelemetry Line Protocol (OTLP) directly in apm-server. You should either use the `otlp` exporter in opentelemetry-collector, pointing it at apm-server, or configure your instrumented applications (OpenTelemetry SDKs) to point directly at apm-server.

> But we are not sure weather to use indices or pipeline to feed into Elasticsearch to achieve our purpose. Nor we could find any relevant documentation or article for such settings.

Indices and ingest pipelines are two very different things -- you don't use one in place of the other. You can read about what indices are at [Data in: documents and indices | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/documents-indices.html), and about ingest pipelines at [Ingest pipelines | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html)

My recommendations are (unless you specifically _need_ Logstash):

1. Configure APM Server to output directly to Elasticsearch. This is the default, most common, and simplest, configuration.
2. Ensure you have at least one Elasticsearch ingest node, and run APM Server with its default pipeline configuration. i.e. leave `output.elasticsearch.pipeline` unspecified.

When you run APM Server in this way, APM Server will install the ingest pipelines itself.

---

<div class="post-metadata">

**Author:** ![nitin194](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitin194/32/92246_2.png) [@nitin194](https://discuss.elastic.co/u/nitin194)\
**Post date:** [July 28, 2021, 10:15am UTC](https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740/5 "2021-07-28T10:15:12Z")

</div>

Hello @axw  
Thank you for your suggestions. It really helps to understand the flow. So we have tried to configured in a similar fashion.

> You should either use the `otlp` exporter in opentelemetry-collector, pointing it at apm-server

```auto
config.yaml: |-
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:55680
          http:
            endpoint: 0.0.0.0:55681
      hostmetrics:
        collection_interval: 1m
        scrapers:
          cpu:
          load:
          memory:
    processors:
      batch: null
    exporters:
      elastic:
        apm_server_url: 'http://apm-server-apm-server.elasticsearch.svc.cluster.local:8200'
        # secret_token: 'XXXXXXXXXXXXXXX'
      logging:
        loglevel: DEBUG
    extensions:
      health_check:
    service:
      pipelines:
        metrics:
          receivers:
            - otlp
            - hostmetrics
          exporters:
            - logging
            - elastic
        traces:
          receivers:
            - otlp
          processors:
            - batch
          exporters:
            - elastic
            - logging

```

> 1. Configure APM Server to output directly to Elasticsearch. This is the default, most common, and simplest, configuration.
> 2. Ensure you have at least one Elasticsearch ingest node, and run APM Server with its default pipeline configuration. i.e. leave `output.elasticsearch.pipeline` unspecified.

```auto
  apm-server.yml: |
    apm-server:
      host: "0.0.0.0:8200"
    queue: {}
    setup.template.settings:
      index:
        number_of_shards: 1
        codec: best_compression
    setup.dashboards.enabled: false
    setup.kibana:
      host: "http://kibana-kibana.elasticsearch.svc.cluster.local"
    output.elasticsearch:
      hosts: ["http://es-client.elasticsearch.svc.cluster.local:9200"]

```

Could you please review above configs for OTEL and APM respectively and suggest? There is no error in logs as of now

Also, how would we be able to see data in APM, Logs & Metrics section under Observability module?

Regards  
Nitin G

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [July 29, 2021, 2:58am UTC](https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740/6 "2021-07-29T02:58:34Z")

</div>

> Could you please review above configs for OTEL and APM respectively and suggest? There is no error in logs as of now

Your APM Server config looks right. I suggest changing your opentelemetry-collector config to use the "otlp" exporter, like in [OpenTelemetry integration | APM User Guide [8.11] | Elastic](https://www.elastic.co/guide/en/apm/get-started/current/open-telemetry-elastic.html#open-telemetry-collector). Something like:

```yaml
exporters:
  otlp/elastic:
    endpoint: "apm-server-apm-server.elasticsearch.svc.cluster.local:8200"
    insecure: true # set this if you're not configuring APM Server with TLS
    headers:
      Authorization: "Bearer <secret_token>"

```

Once you've done that, check the apm-server logs to make sure it is receiving requests, and whether there are any errors in it. It you can't see any requests in the apm-server log, then check the opentelemetry-collector log to make sure it is configured correctly.

> Also, how would we be able to see data in APM, Logs & Metrics section under Observability module?

If your opentelemetry-collector (and SDKs) and apm-server are configured correctly, then data should just start showing up in the APM app in Kibana. If not, then there's likely an issue in one of the above components and you'll need to look through the logs to debug the configuration.

Note that we don't currently support OpenTelemetry's experimental log data. If you want to see logs for your applications, I would recommend using Filebeat to consume logs from Docker or from log files on disk: [Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/index.html)

---

<div class="post-metadata">

**Author:** ![nitin194](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitin194/32/92246_2.png) [@nitin194](https://discuss.elastic.co/u/nitin194)\
**Post date:** [July 29, 2021, 8:21am UTC](https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740/7 "2021-07-29T08:21:26Z")

</div>

Thank you so much for all the details and reference. This helped me create a agent and collector setup. I have marked one of your answers as solution. Though I have more queries. But I will ask them in other thread.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 19, 2021, 4:21am UTC](https://discuss.elastic.co/t/there-are-no-ingest-nodes-in-this-cluster-unable-to-forward-request-to-an-ingest-node/279740/8 "2021-08-19T04:21:43Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.