# Elasticsearch APM Fleet integration - index per application

**URL:** <https://discuss.elastic.co/t/elasticsearch-apm-fleet-integration-index-per-application/372653>\
**Category:** Elastic Observability\
**Created:** [December 31, 2024, 7:52am UTC](https://discuss.elastic.co/t/elasticsearch-apm-fleet-integration-index-per-application/372653 "2024-12-31T07:52:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![fabry\_00](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fabry\_00](https://discuss.elastic.co/u/fabry_00)\
**Post date:** [December 31, 2024, 7:52am UTC](https://discuss.elastic.co/t/elasticsearch-apm-fleet-integration-index-per-application/372653/1 "2024-12-31T07:52:58Z")

</div>

We have an elasticsearch cluster with 4 data nodes. We have our FleetServers with APM integration enabled. We have noticed that elasticsearch creates at least an index per app / service for example:

- .ds-logs-apm.app.myapp1-2024.12.19-000019
- .ds-logs-apm.app.myapp2-2024.12.19-000019
- .ds-logs-apm.app.myapp3-2024.12.19-000019
- .ds-metrics-apm.app.myapp1-2024.12.19-000019
- .ds-metrics-apm.app.myapp2-2024.12.19-000019
- .ds-metrics-apm.app.myapp3-2024.12.19-000019

The result is that we hit our shards limits (4000/4000). We were wondering if we can setup the APM integration to save all the metrics and the logs in two big indexes (one for metrics and one for logs) instead of split them per app.

if yes how to do it and what are the cons?

thank you

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 1, 2025, 7:34pm UTC](https://discuss.elastic.co/t/elasticsearch-apm-fleet-integration-index-per-application/372653/2 "2025-01-01T19:34:26Z")

</div>

Hi @fabry_00

Before we try to help, there is a couple questions.

What version are you on?

Are you using the Managed Fleet Based APM Server or APM Server binary

See [here](https://www.elastic.co/guide/en/observability/current/apm-getting-started-apm-server.html)

And when I looks at the datastream back index...

`.ds-logs-apm.app.myapp1-2024.12.19-000019`

it seems like it is missing the namespace I would expect

`.ds-logs-apm.app.myapp1-default-2024.12.19-000019`

So perhaps provide a bit more detail and we can help... and yes you can "collapse" those data streams ... a few caveats but yes....

If you are in 8.x and using fleet managed you can add a pipeline like this... which will collapse the data streams into a data stream

`logs-apm.app.default-default`

**CAUTION Test First**

```auto
PUT _ingest/pipeline/logs-apm.app@custom
{
  "processors": [
    {
      "set": {
        "if": "ctx.service?.environment == null",
        "field": "service.environment",
        "value": "unknown"
      }
    },
    {
      "set": {
        "field": "custom_pipeline",
        "value": "logs-apm.app@custom"
      }
    },
    {
      "reroute": {
        "dataset": [
          "apm.app.default"
        ],
        "namespace": [
          "default"
        ]
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![fabry\_00](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fabry\_00](https://discuss.elastic.co/u/fabry_00)\
**Post date:** [January 3, 2025, 8:14am UTC](https://discuss.elastic.co/t/elasticsearch-apm-fleet-integration-index-per-application/372653/3 "2025-01-03T08:14:20Z")

</div>

Hi @stephenb ,

We are using ELK 8.16.2  
All our cluster and fleet servers are deployed on premise and are using the Fleet-managed APM Server.

When you say "it seems like it is missing the namespace I would expect", I can tell that we have customized the "namespace" configuration in the APM integration with "prod" following the example here [Data streams | Fleet and Elastic Agent Guide [8.16] | Elastic](https://www.elastic.co/guide/en/fleet/8.16/data-streams.html#data-streams-naming-scheme).  
So, yes the real indexes names are: `.ds-logs-apm.app.myapp1-prod-2024.12.19-000019`

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 3, 2025, 5:37pm UTC](https://discuss.elastic.co/t/elasticsearch-apm-fleet-integration-index-per-application/372653/4 "2025-01-03T17:37:29Z")

</div>

> [@fabry\_00](#):
>
> When you say "it seems like it is missing the namespace I would expect", I can tell that we have customized the "namespace" configuration in the APM integration with "prod" following the example here [Data streams | Fleet and Elastic Agent Guide [8.16] | Elastic](https://www.elastic.co/guide/en/fleet/8.16/data-streams.html#data-streams-naming-scheme).  
> So, yes the real indexes names are: `.ds-logs-apm.app.myapp1-prod-2024.12.19-000019`

Being precise helps.... 😉

> [@fabry\_00](#):
>
> We are using ELK 8.16.2  
> All our cluster and fleet servers are deployed on premise and are using the Fleet-managed APM Server.

Excellent ... so you can try the Pipeline I suggested above just change

```auto
        "namespace": [
          "prod"
        ]

```

and that will put all your APM logs into the single data stream.  
`logs-apm.app.default-prod`  
All the Kibana Apps / functionality / etc... will / should still work.

You can change the dataset to your own name.. just make sure it starts with `apm.app.` example others have done `apm.app.all`

Make sure you test in Non-Prod

Same approach to metrics should work... there can sometimes be field collision so definitely test in in Non-Prod