How to Configure APM in Elastic Cloud to Store Per-Service Indices (Instead of Default Data Streams)

Elastic Observability APM
,
1

Kibana version:
8.11.1 (Elastic Cloud)

Elasticsearch version:
8.11.1 (Elastic Cloud)

APM Server version:
Managed by Elastic Cloud (8.11.1)

APM Agent language and version:
Multiple services using Elastic APM agents (Python, Java)

Browser version:
Latest Chrome

Original install method:
Elastic Cloud – fully managed deployment

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup?

  • APM is **managed entirely by Elastic Cloud
  • We do **not have access to `apm-server.yml
  • All services currently log into the same index:
    .ds-traces-apm-default-*
  • We want to apply different ILM policies per service, so we need separate indices per service (e.g. traces-apm-authservice-*, traces-apm-payment-*)
  • Tried disabling data streams via:
PUT _cluster/settings  
{ "persistent": { "xpack.apm.data_streams.enabled": "false" } }

→ Response:
illegal_argument_exception"`, setting not recognized

Description of the problem including expected versus actual behavior:
We are looking to split APM traces into separate indices per service, in order to apply different ILM policies and control data retention (and cost) more efficiently.

However, since APM is managed in Elastic Cloud:

  • We cannot modify apm-server.yml
  • The UI does not provide an option to change output.elasticsearch.index
  • Data streams seem to be forced, and cannot be disabled via the cluster settings

We’re unsure if this is currently supported in Elastic Cloud or if there’s a supported workaround.

Steps to reproduce:

  1. Deploy services with Elastic APM agents (service.name set)
  2. Observe that all data ends up in .ds-traces-apm-default-*
  3. Try to disable data streams using cluster settings (get error)
  4. No access to apm-server.yml to apply custom index naming

Errors in browser console (if relevant):
N/A

Provide logs and/or server output (if relevant):

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "persistent setting [xpack.apm.data_streams.enabled], not recognized"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "persistent setting [xpack.apm.data_streams.enabled], not recognized"
  },
  "status": 400
}

Question:
Is there any supported way in Elastic Cloud to:

  1. Disable APM data streams?
  2. Apply custom index naming per service (like traces-apm-%{[service.name]})
  3. Or apply separate ILM policies per service?

Would appreciate any suggestions or confirmation if this is possible via UI or API or if it would require a self-managed APM server.

Thanks in advance!