Failed to flush export bulks, Caused by: 413 Request Entity Too Large

Elastic Stack Elasticsearch
1

I have elasticsearch monitoring enabled via setting:
xpack.monitoring.collection.enabled: true, and logstash and kibana, or any other beats sends their monitoring data to the elastic cluster which forwards/exports all the cluster monitoring data to monitoring cluster.

I see the below error in main elasticsearch cluster that sends its monitoring data to a different monitoring cluster:

{"type": "server", "timestamp": "2023-06-06T05:14:21,943Z", "level": "WARN", "component": "o.e.x.m.MonitoringService", "cluster.name": "apm-elastic-lz-02", "node.name": "elasticsearch-master-1", "message": "monitoring execution failed", "cluster.uuid": "4P4zni1TSWi3AgqSI8vK9w", "node.id": "B85_2RvSScSUqVHKm-BJJA" ,
"stacktrace": ["org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks",
"at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:110) [x-pack-monitoring-7.17.1.jar:7.17.1]",
"at org.elasticsearch.xpack.monitoring.exporter.http.HttpExportBulk$1.onFailure(HttpExportBulk.java:148) [x-pack-monitoring-7.17.1.jar:7.17.1]",
"Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://coord.elastic-em.domain.com:80], URI [/_bulk?pipeline=xpack_monitoring_7&filter_path=errors%2Citems.*.error], status line [HTTP/1.1 413 Request Entity Too Large]",
"stacktrace": ["org.elasticsearch.client.ResponseException: method [POST], host [http://coord.elastic-em.domain.com:80], URI [/_bulk?pipeline=xpack_monitoring_7&filter_path=errors%2Citems.*.error], status line [HTTP/1.1 413 Request Entity Too Large]",

Also, Elasticsearch is deployed in an EKS cluster in X account that sends the monitoring data to different monitoring cluster deployed in Y AWS account's EKS cluster.
Route53 endpoint -> nginx ingress controller (loadbalancer) -> coord pods.

I have modified the below setting in nginx
large-client-header-buffers: 4 128k. This didnt work.

I see that the error says the request is too large. From my readings, the default max content length is 100mb. -> not exactly sure what this means.

Any suggestions are appreciated.

2

How often does this happen?

3

its happening continuosly, no monitoring data for this cluster getting into the monitoring cluster.
while other clusters which are much bigger in size are able to send their monitoring data.

4

What is the output from the _cluster/stats?pretty&human API?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.