Hi,
We are running a metricbeat daemon set within two GKE environments, a test and production environment. Both metricbeats are sending data to the same elasticsearch monitoring cluster. De metricbeat from the test environment is configured to output data to a custom index, see config below.
Config test metricbeat:
setup.template.name: "test-metricbeat"
setup.template.pattern: "test-metricbeat-*"setup.template.settings:
index.number_of_shards: 1
index.codec: best_compressionsetup.ilm.enabled: auto
setup.ilm.rollover_alias: "test-metricbeat"
setup.ilm.pattern: "{now/d}-000001"setup.kibana:
host: '${KIBANA_HOST}'output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
index: "test-metricbeat-%{[agent.version]}-%{+yyyy.MM.dd}"
If we only deploy metricbeat to the production environment everything works correct, but deploying the second metricbeat to the test enviroment gives the following errors:
"2020-02-17T16:39:37.575Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure
2020-02-18 15:08:46.981 CET
2020-02-18T14:08:46.981Z ERROR elasticsearch/client.go:344 Failed to perform any bulk index operations: 429 Too Many Requests: {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1015096192/968mb], which is larger than the limit of [1003493785/957mb], real usage: [1014940576/967.9mb], new bytes reserved: [155616/151.9kb], usages [request=0/0b, fielddata=58922011/56.1mb, in_flight_requests=2334824/2.2mb, accounting=29581323/28.2mb]","bytes_wanted":1015096192,"bytes_limit":1003493785,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1015096192/968mb], which is larger than the limit of [1003493785/957mb], real usage: [1014940576/967.9mb], new bytes reserved: [155616/151.9kb], usages [request=0/0b, fielddata=58922011/56.1mb, in_flight_requests=2334824/2.2mb, accounting=29581323/28.2mb]","bytes_wanted":1015096192,"bytes_limit":1003493785,"durability":"PERMANENT"},"status":429}
Elasticsearch monitorings cluster runs on a elastic cloud solution, two nodes (hot/warm). Do i need to adjust some limits to solve the issues?