still looking at data but so far it's looking like setting allow_custom_routing to false fixes the issue. And this is without adding the 2 roles to the nodes that you had mentioned above so this confirms they are not required, even for datastreams.
I am curious why this is being set in the default metricbeat template and why it's causing issues for us and presumably not for everyone else that is using it. Should I file a bug with metricbeat about this?
Good to Know... BUT I would expect all Beats, Agent, etc integrations will have that setting so at some point getting the right roles in would probably make sense....
data_hot etc are the new standard...
adding the roles did not fix the issue, I tried and it made no difference.
You rebooted the nodes? with the correct settings?
Ok and wierd.... I just created a single node cluster 8.14.3
with
"roles": [
"master",
"data"
]
},
Default metricbeat is ingesting fine
All
GET _cluster/allocation/explain?include_yes_decisions=true
{
"index": ".ds-metricbeat-8.14.1-2024.08.28-000001",
"shard": 0,
"primary": true
}
and explain says yes...
{
"index": ".ds-metricbeat-8.14.1-2024.08.28-000001",
"shard": 0,
"primary": true,
"current_state": "started",
"current_node": {
"id": "-HcQrNwCTbG_l0eL3LcAPQ",
"name": "f0d64ac033e4",
"transport_address": "172.18.0.2:9300",
"attributes": {
"transform.config_version": "10.0.0",
"xpack.installed": "true",
"ml.config_version": "12.0.0"
},
"roles": [
"data",
"master"
],
"weight_ranking": 1
},
so call me confused too... let me think a bit more
@DavidTurner Sorry to both er you ... but I am confused too why
This user running metricbeat with default template is getting this error
{\"type\":\"routing_missing_exception\",\"reason\":\"routing is required for [.ds-dev2-cust-eastus-metrics-8.14.0-2024.08.27-2024.08.27-000001
Using the Default metricbeat template with node roles
"roles": [
"master",
"data"
]
},
first I though the data stream required data_hot so would not work with just data but that does not seem to be the case.... that works... so not sure why the error.
The exception means that this index's mapping requires routing. I'd be surprised if that requirement comes from Metricbeat's default templates, but I'm not familiar enough with those templates to say for sure.
the index does require routing however everything works fine with that until applying the metricbeat index template.
When I see this data stream name...
".ds-dev2-cust-eastus-metrics-8.14.0-2024.08.28-2024.08.28-000001":
With the double date etc... it makes me think that perhaps the index template you think is getting applied is actually getting applied....
so question, if you just run metricbeat with no changes at all to anything... just let it write the system metrics to the data stream it wants to... does it work?
Just OOTB metricbeat just change the output to your elasicsearch run setup and then run... does it work?
It's using the correct index template, I can see that when I look in the kibana UI for the data stream, it tells you which index template it's using:
the dev2-metrics is the index template I have been modifying to either use or not use the provided JSON from the metricbeat template.
I'm not actually sure where the double date is coming from though, you can see it's not that way in the datasource name but for some reason the indexes are getting named with the double date. I assumed this was an elasticsearch thing.
And this ^^^ we need to eliminate a variable .... can you try that...
I usually find that there is something non-obvious... that will become obvious 
Did you change any cluster level routing settings?
GET _cluster/settings
{
"persistent": {
"xpack": {
"monitoring": {
"elasticsearch": {
"collection": {
"enabled": "true"
}
},
"collection": {
"enabled": "true"
}
}
}
},
"transient": {
"cluster": {
"routing": {
"allocation": {
"cluster_concurrent_rebalance": "20"
}
}
},
"xpack": {
"monitoring": {
"elasticsearch": {
"collection": {
"enabled": "true"
}
},
"collection": {
"enabled": "true"
}
}
}
}
}
working on trying out the metricbeat setup now, we have not been using that because this is all build off of playbooks and it's easier to handle managing the index templates in the playbook rather than automating connecting to a pod and running the setup command
ran into some issues with conflicting index templates so I lowered the priority on those and now I'm getting this error from metricbeat setup:
{"log.level":"error","@timestamp":"2024-08-30T17:56:48.413Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1345},"message":"Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"},\"status\":400}. Response body: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"no matching index template found for data stream [platdev2-metrics]\"},\"status\":400}","service.name":"metricbeat","ecs.version":"1.6.0"}
isn't this supposed to be creating that index template?
edit: even though it gave that error it does appear that it created the platdev2-metrics index template
ok, this is interesting. so...this is now working using the index templated loaded from metricbeat setup BUT there is a difference in this template vs the one I had been trying to use. It has an empty data stream section:
"data_stream": {}
vs
"data_stream": {
"hidden": false,
"allow_custom_routing": true
},
which was in the one that I was first trying to use which came from metricbeat somewhere but I cannot find where, through the process of troubleshooting I've re-installed the metricbeat playbooks and now this is the template I'm getting from both my test and dev metricbeat pods. Was this allow_custom_routing setting being applied on older versions of the metricbeat index template?
Either way, it does seem to come back to this allow_custom_routing setting on the datastream being the issue.
EDIT ... Think we crossed int the internets.....
Apologies perhaps I was not clear... I was hoping you would run metricbeat without any conflict or other matching templates.... the fact you are have multiple template with priorities and multiple matching templates etc... I think is masking / confusing the issues ...
I would expect if you just ran the metricbeat with the defaults.... no changes to metricbeat.yml except host name and no other matching templates is that you would get a data stream like
metricbeat-8.14.3
not the
platdev2-metrics
The fact you are not getting metricbeat-8.14.3 means other forces / templates are at play...
I Can't really help debug without that test ....
I think this is coming from the settings we are applying to metricbeat. In our config we are specifying:
setup:
template:
enabled: true
name: "${INDEX_TEMPLATE_NAME}"
pattern: "*-*-*-metrics-*"
oh, actually does that tell MB to try to run setup every time it runs? I think it might.
anyways, it does look like the problem is still this allow_custom_routing which at this point I don't know which version of metricbeat I was using when I ran the command to output the JSON for the template but it seems it's not there on the 8.14.0 pods that I have.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.