# Unable to ship metrics when using metricbeat provided index template, 400 error "routing\_missing\_exception" **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [August 27, 2024, 6:54pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638 "2024-08-27T18:54:11Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 27, 2024, 6:54pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/1 "2024-08-27T18:54:11Z")
Hello, We are using metricbeat (8.14) on k8s to ship metrics to an elasticsearch (8.13) cluster. This has been working well however last week we discovered a mapping issue where sometimes some of our metrics are getting set as a float and somtimes they are getting set as a long. We have been using dynamic mapping and this is likely why this is happening. We would like to move to static mapping using the metricbeat provided index template. Our current index template looks like so: ```auto { "template": { "settings": { "index": { "lifecycle": { "name": "bmap_obsv_ilm_policy" }, "routing": { "allocation": { "include": { "_tier_preference": "data_hot" } } }, "mapping": { "total_fields": { "limit": "2000" } }, "refresh_interval": "30s", "number_of_shards": "1", "number_of_replicas": "1" } }, "mappings": { "dynamic": "true" }, "aliases": {} } } ``` On our test environment I was able to switch to the metricbeat provided index template and everything worked as expected however when I switched our dev environment over to it I was getting the following error from metricbeat: ```auto {"log.level":"debug","@timestamp":"2024-08-27T16:49:15.747Z","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).bulkCollectPublishFails","file.name":"elasticsearch/client.go","file.line":430},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Meta:null, Fields:null, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}, EncodedEvent:(*elasticsearch.encodedEvent)(0xc004c26d80)} (status=400): {\"type\":\"routing_missing_exception\",\"reason\":\"routing is required for [.ds-dev2-cust-eastus-metrics-8.14.0-2024.08.27-2024.08.27-000001]/[noC9lJEBUqYkhwlM2TDh]\",\"index_uuid\":\"_na_\",\"index\":\".ds-dev2-cust-eastus-metrics-8.14.0-2024.08.27-2024.08.27-000001\"}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"} ``` I'm looking for help trying to figure out why this might be. Here is the index template which we are trying to move to but having issues: (too large to include here) > switching back to the current basic dynamic index template fixes the issue and allows metricbeat to start shipping metrics again and this index template is all that is changing so it seems something specified in this index template is causing issues but I'm unsure what. ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 27, 2024, 6:54pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/2 "2024-08-27T18:54:59Z")
If anyone help explain why this would work fine in one environment of ours but cause the 400 errors and how to fix those that would be much appreciated. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 28, 2024, 3:35am UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/3 "2024-08-28T03:35:13Z")
On the env that is not working... > [@dfinn](#): > > `.ds-dev2-cust-eastus-metrics-8.14.0-2024.08.27-2024.08.27-000001` Hi @dfinn You need to show us all the index settings for the index that is throwing the errors `GET .ds-dev2-cust-eastus-metrics-8.14.0-2024.08.27-2024.08.27-000001/_settings` Then `GET _nodes` and show the `roles` and `routing` settings from same env... ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 5:16pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/4 "2024-08-28T17:16:39Z")
ok @stephenb , first off I made a mistake at some point and thought this was working in our test environment but we have issues there as well. at least it's consistent across our environments. here is the output you asked for index settings for today: > node output: > as far as I can tell, all the routing settings look fine. it's looking for `data_hot` and I have nodes that have the `data` role which should be satisfying that. ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 5:18pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/5 "2024-08-28T17:18:50Z")
also including my metricbeat output section in case that is helpful: ```auto output.elasticsearch: hosts: "[${ELASTICSEARCH_HOST}:9200]" protocol: "${ELASTICSEARCH_PROTOCOL:https}" username: "${ELASTICSEARCH_USERNAME:elastic}" ssl.verification_mode: "${ELASTICSEARCH_VERIFYSSL:none}" password: "${ELASTICSEARCH_PASSWORD:welcome1}" index: "${INDEX_NAME}-%{[agent.version]}-%{+yyyy.MM.dd}" allow_older_versions: true ``` ```auto env|grep INDEX_NAME INDEX_NAME=platdev2-cust-eastus-metrics ``` ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 5:19pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/6 "2024-08-28T17:19:44Z")
also, I upgraded our test environment to 8.14.3 to see if that changed anything and it has not, still having the same issue when applying the metricbeat index template ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 28, 2024, 6:07pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/7 "2024-08-28T18:07:50Z")
hi @dfinn Pretty You are missing the correct roles on your nodes... Your index ```auto "routing": { "allocation": { "include": { "_tier_preference": "data_hot" <<< Where it will go... } } }, ``` You nodes.. ```auto "roles": [ <<< NOT HERE , nowhere for index to go.. "data", "ingest" ], ``` should be... ```auto "roles": [ "ingest", "transform", "data_hot", <<< MISSING : Where your index wants to go "data_content", <<< MISSING "master", "remote_cluster_client" ] }, ``` see here > **[Nodes | Elasticsearch Guide \[8.15\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles)** ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 6:10pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/8 "2024-08-28T18:10:51Z")
that is interesting because this works perfectly fine before applying the metricbeat index template. what is it about that template that would require this change? I'm looking for where I saw this but yesterday I found somewhere in the elasticsearch documentation that having the data role on a node satisfies the data\_hot requirement. Trying to find where I saw that... here it is: > **[Nodes | Elasticsearch Guide \[8.15\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html)** " [Data node](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#data-node) A node that has one of several data roles. Data nodes hold data and perform data related operations such as CRUD, search, and aggregations. A node with a generic `data` role can fill any of the specialized data node roles." and lower down on that same page: " If you want to include a node in all tiers, or if your cluster does not use multiple tiers, then you can use the generic `data` role." so I'm not sure this is actually required and it does seem to work fine prior to applying this metricbeat index template. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 28, 2024, 6:27pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/9 "2024-08-28T18:27:30Z")
> **[Nodes | Elasticsearch Guide \[8.15\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#data-hot-node)** > The hot tier is required. New indices that are part of a [data stream](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html) are automatically allocated to the hot tier. > > To create a dedicated hot node, set: > > node.roles: [data\_hot] It is a data stream... it needs the correct roles... Perhaps I am wrong... but they need to match.... ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 6:31pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/10 "2024-08-28T18:31:02Z")
From what I'm reading we should be satisfying the requirement and it does work as expected before applying the MB template, metrics come in and are written to the data stream so I'm nearly sure this is not required. I'm happy to test though, is there a way to modify the roles of an existing node, so far I'm not finding a way to do that. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 28, 2024, 6:33pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/11 "2024-08-28T18:33:53Z")
> [@dfinn](#): > > expected before applying the MB template. Unclear what this means ... what do you mean before applying the template? What was it used before ... are you upgrading from 7.X or something... > [@dfinn](#): > > From what I'm reading we should be satisfying the requirement Yeah, I think the docs can be a bit confusing... data streams require the correct roles... Metricbeat 8.x uses data streams Data streams require the correct roles > [@dfinn](#): > > I'm happy to test though, is there a way to modify the roles of an existing node, so far I'm not finding a way to do that > **[Nodes | Elasticsearch Guide \[8.15\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles)** > You define a node’s roles by setting `node.roles` in `elasticsearch.yml` . ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 6:38pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/12 "2024-08-28T18:38:22Z")
Sorry, I mentioned this in the original post but maybe it wasn't clear. Prior to trying to use the specific metricbeat index template we have been using a very basic index template which uses dynamic mapping and this has been working perfectly fine for us other than occasionally running into issues where field types get set incorrectly, hence the need to use the metricbeat index template if possible. Here is our current/old index template for metricbeat datastreams: ```auto { "template": { "settings": { "index": { "lifecycle": { "name": "bmap_obsv_ilm_policy" }, "routing": { "allocation": { "include": { "_tier_preference": "data_hot" } } }, "mapping": { "total_fields": { "limit": "2000" } }, "refresh_interval": "30s", "number_of_shards": "1", "number_of_replicas": "1" } }, "mappings": { "dynamic": "true" }, "aliases": {} } } ``` pretty simple but as you can see it has the same routing settings specified. When using this index template everything works as expected, metricbeat can ship metrics and they get written to the correct datastream. the issue arises when we switch to the metricbeat specific index template and it's not clear why that is causing issues. This happens on both 8.13.0 and 8.14.3. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 28, 2024, 6:42pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/13 "2024-08-28T18:42:54Z")
Probably because your previous was not a data stream, just a plain ole index... so the node role was not enforced... Are you sure that was a data stream? That previous template does not appear to be a data stream... I am not sure at this point... but there is an easy test... fix your node roles.... ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 6:46pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/14 "2024-08-28T18:46:33Z")
yes, it's previously a datastream and has been that way for quite some time. I'm testing now, attempting to add the 2 roles you mentioned to our data nodes. If it helps I could provide settings from our dev cluster which at this point I've rolled back to using the dynamic mapping template for metricbeat datastreams. edit, here is the settings from todays metricbeat datastream from our dev cluster using the dynamic mapping template, you can see it's a datastream and has the same routing settings as our test env: ```auto { ".ds-dev2-cust-eastus-metrics-8.14.0-2024.08.28-2024.08.28-000001": { "settings": { "index": { "lifecycle": { "name": "bmap_obsv_ilm_policy" }, "routing": { "allocation": { "include": { "_tier_preference": "data_hot" } } }, "mapping": { "total_fields": { "limit": "2000" } }, "refresh_interval": "30s", "hidden": "true", "number_of_shards": "1", "provided_name": ".ds-dev2-cust-eastus-metrics-8.14.0-2024.08.28-2024.08.28-000001", "creation_date": "1724803201210", "number_of_replicas": "1", "uuid": "EZWofRcYROeXP-o4StyOVw", "version": { "created": "8503000" } } } } } ``` and the roles in this environment are the same (which makes sense, this is all being deployed via playbooks so it should be identical): ```auto "roles": [ "data", "ingest" ], ``` so, if there is a requirement to add these additional roles that requirement does not arise until applying the metricbeat specific index template ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 28, 2024, 6:55pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/15 "2024-08-28T18:55:56Z")
If you look deeply into the metricbeat template you will see this... ```auto "data_stream": { "hidden": false, "allow_custom_routing": false <<< Might be this setting which is requiring the data hot... } } ``` It may be AHHH I am missing something.... your is a data stream ` ".ds-dev2-cust-eastus....` ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 6:57pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/16 "2024-08-28T18:57:44Z")
yes, nearly positive it's a data stream, it shows up in the data stream section in the kibana UI ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 28, 2024, 6:58pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/17 "2024-08-28T18:58:21Z")
it is probably the ` "allow_custom_routing": false` \<\<\< Might be this setting which is requiring the data hot... You could try changing that... in the template... then restart without reloading the template. ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 6:59pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/18 "2024-08-28T18:59:29Z")
ok, interesting. the `allow_custom_routing` option comes from the default MB template and I just left it, I can try removing. But...data is now flowing into that datastream after adding those 2 roles. So maybe something about the `allow_custom_routing` setting is what is requiring those roles to be specifically applied to the nodes? what is the proper fix here? add these roles or set `allow_custom_routing` to false? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 28, 2024, 7:08pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/19 "2024-08-28T19:08:44Z")
This is good thanks for the Discussion ... I did not see customer routing before... Today I Learned (TIL) ---
**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Post date:** [August 28, 2024, 7:12pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/20 "2024-08-28T19:12:06Z")
actually, I may have spoke too soon. I thought I saw that there was now documents in this index but there are not any after further inspection. I'm testing now with setting `allow_custom_routing` to false but adding the roles to the nodes did not fix the issue. [Next page](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638.md?page=2)