# Unable to ship metrics when using metricbeat provided index template, 400 error "routing\_missing\_exception"

**URL:** <https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [August 27, 2024, 6:54pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638 "2024-08-27T18:54:11Z")\
**Posts on this page:** 15\
**Page:** 2

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [August 28, 2024, 7:50pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/21 "2024-08-28T19:50:40Z")

</div>

still looking at data but so far it's looking like setting `allow_custom_routing` to `false` fixes the issue. And this is without adding the 2 roles to the nodes that you had mentioned above so this confirms they are not required, even for datastreams.

I am curious why this is being set in the default metricbeat template and why it's causing issues for us and presumably not for everyone else that is using it. Should I file a bug with metricbeat about this?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 28, 2024, 8:55pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/22 "2024-08-28T20:55:29Z")

</div>

> [@dfinn](#):
>
> still looking at data but so far it's looking like setting `allow_custom_routing` to `false` fixes the issue. And this is without adding the 2 roles to the nodes that you had mentioned above so this confirms they are not required, even for datastreams.

Good to Know... BUT I would expect all Beats, Agent, etc integrations will have that setting so at some point getting the right roles in would probably make sense....

`data_hot` etc are the new standard...

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [August 28, 2024, 9:07pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/23 "2024-08-28T21:07:14Z")

</div>

adding the roles did not fix the issue, I tried and it made no difference.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 28, 2024, 10:34pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/24 "2024-08-28T22:34:24Z")

</div>

You rebooted the nodes? with the correct settings?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 28, 2024, 11:05pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/25 "2024-08-28T23:05:14Z")

</div>

Ok and wierd.... I just created a single node cluster 8.14.3

with

```auto
          "roles": [
            "master",
            "data"
          ]
        },

```

Default metricbeat is ingesting fine

All

```auto
GET _cluster/allocation/explain?include_yes_decisions=true
{
  "index": ".ds-metricbeat-8.14.1-2024.08.28-000001",
  "shard": 0,
  "primary": true
}

```

and explain says yes...

```auto
{
  "index": ".ds-metricbeat-8.14.1-2024.08.28-000001",
  "shard": 0,
  "primary": true,
  "current_state": "started",
  "current_node": {
    "id": "-HcQrNwCTbG_l0eL3LcAPQ",
    "name": "f0d64ac033e4",
    "transport_address": "172.18.0.2:9300",
    "attributes": {
      "transform.config_version": "10.0.0",
      "xpack.installed": "true",
      "ml.config_version": "12.0.0"
    },
    "roles": [
      "data",
      "master"
    ],
    "weight_ranking": 1
  },

```

so call me confused too... let me think a bit more

@DavidTurner Sorry to both er you ... but I am confused too why  
This user running metricbeat with default template is getting this error

```auto
{\"type\":\"routing_missing_exception\",\"reason\":\"routing is required for [.ds-dev2-cust-eastus-metrics-8.14.0-2024.08.27-2024.08.27-000001

```

Using the Default metricbeat template with node roles

```auto
          "roles": [
            "master",
            "data"
          ]
        },

```

first I though the data stream required `data_hot` so would not work with just `data` but that does not seem to be the case.... that works... so not sure why the error.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 29, 2024, 7:09pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/26 "2024-08-29T19:09:42Z")

</div>

The exception means that this index's mapping [requires routing](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html#_making_a_routing_value_required). I'd be surprised if that requirement comes from Metricbeat's default templates, but I'm not familiar enough with those templates to say for sure.

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [August 30, 2024, 2:17pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/27 "2024-08-30T14:17:47Z")

</div>

the index does require routing however everything works fine with that until applying the metricbeat index template.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 30, 2024, 3:18pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/28 "2024-08-30T15:18:29Z")

</div>

@dfinn

When I see this data stream name...

`".ds-dev2-cust-eastus-metrics-8.14.0-2024.08.28-2024.08.28-000001":`  
With the double date etc... it makes me think that perhaps the index template you think is getting applied is actually getting applied....

> [@dfinn](#):
>
> the index does require routing however everything works fine with that until applying the metricbeat index template.

so question, if you just run metricbeat with no changes at all to anything... just let it write the system metrics to the data stream it wants to... does it work?

Just OOTB metricbeat just change the output to your elasicsearch run setup and then run... does it work?

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [August 30, 2024, 3:33pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/29 "2024-08-30T15:33:52Z")

</div>

It's using the correct index template, I can see that when I look in the kibana UI for the data stream, it tells you which index template it's using:

 ![Index_Management_-_Elastic](https://us1.discourse-cdn.com/elastic/original/3X/5/c/5c75a321c11522b46652bdae4ad0c86cf7568ab4.jpeg)

the `dev2-metrics` is the index template I have been modifying to either use or not use the provided JSON from the metricbeat template.

I'm not actually sure where the double date is coming from though, you can see it's not that way in the datasource name but for some reason the indexes are getting named with the double date. I assumed this was an elasticsearch thing.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 30, 2024, 5:18pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/30 "2024-08-30T17:18:09Z")

</div>

> [@dfinn](#):
>
> the `dev2-metrics` is the index template I have been modifying to either use or not use the provided JSON from the metricbeat template.

> [@stephenb](#):
>
> so question, if you just run metricbeat with no changes at all to anything... just let it write the system metrics to the data stream it wants to... does it work?
> 
> Just OOTB metricbeat just change the output to your elasicsearch run setup and then run... does it work?

And this ^^^ we need to eliminate a variable .... can you try that...

I usually find that there is something non-obvious... that will become obvious 🙂

Did you change any cluster level routing settings?

`GET _cluster/settings`

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [August 30, 2024, 5:30pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/31 "2024-08-30T17:30:51Z")

</div>

> [@stephenb](#):
>
> GET \_cluster/settings

```auto
{
  "persistent": {
    "xpack": {
      "monitoring": {
        "elasticsearch": {
          "collection": {
            "enabled": "true"
          }
        },
        "collection": {
          "enabled": "true"
        }
      }
    }
  },
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "cluster_concurrent_rebalance": "20"
        }
      }
    },
    "xpack": {
      "monitoring": {
        "elasticsearch": {
          "collection": {
            "enabled": "true"
          }
        },
        "collection": {
          "enabled": "true"
        }
      }
    }
  }
}

```

working on trying out the metricbeat setup now, we have not been using that because this is all build off of playbooks and it's easier to handle managing the index templates in the playbook rather than automating connecting to a pod and running the setup command

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [August 30, 2024, 5:58pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/33 "2024-08-30T17:58:41Z")

</div>

ran into some issues with conflicting index templates so I lowered the priority on those and now I'm getting this error from `metricbeat setup`:

```auto
{"log.level":"error","@timestamp":"2024-08-30T17:56:48.413Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1345},"message":"Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"},\"status\":400}. Response body: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"},\"status\":400}","service.name":"metricbeat","ecs.version":"1.6.0"}

```

isn't this supposed to be creating that index template?

edit: even though it gave that error it does appear that it created the `platdev2-metrics` index template

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [August 30, 2024, 6:13pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/34 "2024-08-30T18:13:09Z")

</div>

ok, this is interesting. so...this is now working using the index templated loaded from `metricbeat setup` BUT there is a difference in this template vs the one I had been trying to use. It has an empty data stream section:

```auto
"data_stream": {}

```

vs

```auto
   "data_stream": {
     "hidden": false,
     "allow_custom_routing": true
   },

```

which was in the one that I was first trying to use which came from metricbeat somewhere but I cannot find where, through the process of troubleshooting I've re-installed the metricbeat playbooks and now this is the template I'm getting from both my test and dev metricbeat pods. Was this `allow_custom_routing` setting being applied on older versions of the metricbeat index template?

Either way, it does seem to come back to this `allow_custom_routing` setting on the datastream being the issue.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 30, 2024, 6:14pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/35 "2024-08-30T18:14:30Z")

</div>

EDIT ... Think we crossed int the internets.....

> [@dfinn](#):
>
> ran into some issues with conflicting index templates so I lowered the priority on those and now I'm getting this error from `metricbeat setup`:
> 
> ```auto
> {"log.level":"error","@timestamp":"2024-08-30T17:56:48.413Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1345},"message":"Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"},\"status\":400}. Response body: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"},\"status\":400}","service.name":"metricbeat","ecs.version":"1.6.0"}
> 
> ```
> 
> isn't this supposed to be creating that index template?
> 
> edit: even though it gave that error it does appear that it created the `platdev2-metrics` index template

Apologies perhaps I was not clear... I was hoping you would run metricbeat without any conflict or other matching templates.... the fact you are have multiple template with priorities and multiple matching templates etc... I think is masking / confusing the issues ...

I would expect if you just ran the metricbeat with the defaults.... no changes to metricbeat.yml except host name **and no other matching templates is that you would get a data stream like**

`metricbeat-8.14.3`

not the  
`platdev2-metrics`

The fact you are not getting `metricbeat-8.14.3` means other forces / templates are at play...

I Can't really help debug without that test ....

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [August 30, 2024, 6:20pm UTC](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638/36 "2024-08-30T18:20:36Z")

</div>

I think this is coming from the settings we are applying to metricbeat. In our config we are specifying:

```auto
     setup:
       template:
         enabled: true
         name: "${INDEX_TEMPLATE_NAME}"
         pattern: "*-*-*-metrics-*"

```

oh, actually does that tell MB to try to run setup every time it runs? I think it might.

anyways, it does look like the problem is still this `allow_custom_routing` which at this point I don't know which version of metricbeat I was using when I ran the command to output the JSON for the template but it seems it's not there on the 8.14.0 pods that I have.

[Previous page](https://discuss.elastic.co/t/unable-to-ship-metrics-when-using-metricbeat-provided-index-template-400-error-routing-missing-exception/365638.md?page=1)
