Indexing Traces into Separate Indices Based on Application Names or environment

Elastic Observability APM
1

Hi @axw @felixbarny

I followed this issue for a month now and i also read following threads
https://www.elastic.co/guide/en/observability/current/apm-data-streams.html
https://github.com/elastic/apm-data/pull/201

from this what i understand is i can either use reroute processor on custom ingest pipeline to use service.namespace or data_stream.dataset or data_stream.namespace to segregate traces.

however another elastic forum post says from version 8.13 it will happen automatically if i have data_stream.dataset and data_stream.namespace defined in opentelemetry collector resource attributes, So i upgrade our test environment to 8.13 version but i do not see traces being seperately indexes to different data streams.
[https://discuss.elastic.co/t/indexing-application-trace-data-into-separate-indices-based-on-application-names/353437](url)

also the attributes data_stream.dataset and data_stream.namespace i passed from opentelemetry collector were getting added through collector and apm server. however traces are going to traces-apm-default index (not datastream) and data_stream.dataset and data_stream.namespace get renamed to labels.data_stream_dataset and labels.data_stream_namespace.

{
"_index": "traces-apm-default",
"_id": "jbAQwZABcWl2AwZkmZ77",
"_version": 1,
"_score": 0,
"_source": {
"@timestamp": "2024-07-17T14:20:05.966Z",
"agent": {
"name": "otlp",
"version": "unknown"
},
"data_stream": {
"dataset": "apm",
"namespace": "default",
"type": "traces"
},
"event": {
"duration": 6600,
"outcome": "success"
},
"labels": {
"Min": "40",
"Max": "288",
"Name": "4.2.0",
"Current": "40",
"data_stream_dataset": "eastus",
"plan_id": "669028d75ab8ff7cf71bf98f",
"site": "eastus",
"plan_name": "df-rel-01",
"data_stream_namespace": "eastus"
},
"observer": {
"hostname": "elastic-apm-765c57ffdf-9ml55",
"type": "apm-server",
"version": "8.12.0"
},
"service": {
"framework": {
"name": "DI"
},
"language": {
"name": "unknown"
},
"name": "DI Server",
"node": {
"name": "43a024dc-cfd9-4fa4-844c-af19bfdfbe0c"
}
},
"span": {
"id": "44b3f17f4b7da1a0"
},
"timestamp": {
"us": 1721226005966615
},
"trace": {
"id": "553ca8a5033f4a8f86e809b6a05cb744"
},
"transaction": {
"id": "44b3f17f4b7da1a0",
"name": "RequestInspection",
"representative_count": 1,
"result": "Success",
"sampled": true,
"type": "unknown"
}
}

i also tried to create new apm integration to add namespace i am expecting and also created custom ingest pipeline with following specs but that did not help.

{
"reroute": {
"dataset": [
"{{data_stream.dataset}}"
],
"namespace": [
"{{data_stream.namespace}}"
]
}

i will appreciate if you can help me here to fix it?

Kibana version: 8.13

Elasticsearch version: 8.13

APM Server version: 8.13

APM Agent language and version: Opentelemetry collector contrib 0.82.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Deployed on Kubernetes - AKS

Fresh install or upgraded from other version? : Fresh

Steps to reproduce:

  1. Launch OTEL Application, set env variable OTEL_RESOURCE_ATTRIBUTES
  2. Configure to ship to APM Server
  3. Everything goes to the same datastream