Metricbeat - Bulk send failure - 429 Too Many Requests

Hi,

We are running a metricbeat daemon set within two GKE environments, a test and production environment. Both metricbeats are sending data to the same elasticsearch monitoring cluster. De metricbeat from the test environment is configured to output data to a custom index, see config below.

Config test metricbeat:

setup.template.name: "test-metricbeat"
setup.template.pattern: "test-metricbeat-*"

setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "test-metricbeat"
setup.ilm.pattern: "{now/d}-000001"

setup.kibana:
host: '${KIBANA_HOST}'

output.elasticsearch:
hosts: ['{ELASTICSEARCH_HOST}'] username: {ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
index: "test-metricbeat-%{[agent.version]}-%{+yyyy.MM.dd}"

If we only deploy metricbeat to the production environment everything works correct, but deploying the second metricbeat to the test enviroment gives the following errors:

"2020-02-17T16:39:37.575Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure

2020-02-18 15:08:46.981 CET
2020-02-18T14:08:46.981Z ERROR elasticsearch/client.go:344 Failed to perform any bulk index operations: 429 Too Many Requests: {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1015096192/968mb], which is larger than the limit of [1003493785/957mb], real usage: [1014940576/967.9mb], new bytes reserved: [155616/151.9kb], usages [request=0/0b, fielddata=58922011/56.1mb, in_flight_requests=2334824/2.2mb, accounting=29581323/28.2mb]","bytes_wanted":1015096192,"bytes_limit":1003493785,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1015096192/968mb], which is larger than the limit of [1003493785/957mb], real usage: [1014940576/967.9mb], new bytes reserved: [155616/151.9kb], usages [request=0/0b, fielddata=58922011/56.1mb, in_flight_requests=2334824/2.2mb, accounting=29581323/28.2mb]","bytes_wanted":1015096192,"bytes_limit":1003493785,"durability":"PERMANENT"},"status":429}

Elasticsearch monitorings cluster runs on a elastic cloud solution, two nodes (hot/warm). Do i need to adjust some limits to solve the issues?

It looks like you need to increase the size of your hot tier.

I 've changed the hot tier setting to a two node solution.
I still get the same errors and the CPU of one (hot) instance spikes to 100%.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.