# APM Labels: best practices

**URL:** https://discuss.elastic.co/t/apm-labels-best-practices/234924
**Category:** APM
**Created:** [May 29, 2020, 1:23pm UTC](https://discuss.elastic.co/t/apm-labels-best-practices/234924 "2020-05-29T13:23:31Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Emanuel\_Velzi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emanuel_velzi/32/66975_2.png) [@Emanuel\_Velzi](https://discuss.elastic.co/u/Emanuel_Velzi)
#### Post date: [May 29, 2020, 1:23pm UTC](https://discuss.elastic.co/t/apm-labels-best-practices/234924/1 "2020-05-29T13:23:32Z")

</div>

Hi!

I'm using Elastic Stack for Application Performance Management ( **APM** ).  
Many applications will send metrics and **transactions** to a centralised solution of APM.  
I have some **doubts about custom "labels"** , which is a **dynamic** field in the mapping.

These apps don't have so much traffic, so I've created only **one index of transactions** for all apps: together will index about 1TB per day.

However, these apps could index a lot of different "labels" and I fear a mapping explosion.  
**1) How should I manage the possible case of mapping explosion?**

Also, some apps could index labels in a different way than others, I mean: **same name, different type**.  
In this case, ES will have a lot of parse errors. I've tested how ES performs with that errors, and I saw that ES performs about **40% less** than with correct events.

**2) How should I manage this? I would like to index the rest of the document without that invalid labels, is it possible?**  
I've tested the `ignore_malformed` parameter but it not works with booleans for example.

**3) How could I ensure that ES does not degrade so much its performance in case of parsing errors?**

---

<div class="post-metadata">

### Author: ![jalvz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jalvz/32/45613_2.png) [@jalvz](https://discuss.elastic.co/u/jalvz)
#### Post date: [June 3, 2020, 11:17am UTC](https://discuss.elastic.co/t/apm-labels-best-practices/234924/2 "2020-06-03T11:17:24Z")

</div>

Hello there,

First, on the mapping explosion: since you have domain knowledge of the services/apps setting labels, the best way forward is to avoid it in the first place: just not create too many labels, or don't use variables with high cardinality for labels.

If possible, make it so that the apps use the same types for the same labels, or have them create labels with different names so they can not clash (for instance, prefixing the label with the app name).

`ignore_malformed` should work. But keep in mind that it is a template setting, and it won't be applied to existing indices, could be that the reason why it looks it is not working for you? Can you check on newly created indices (after the template update)?

Other alternative would be to have different indices for different services, and then configure ILM policies so that you don't end up with very small daily indices.

Hope this helps, let me know if otherwise it doesn't!

---

<div class="post-metadata">

### Author: ![Emanuel\_Velzi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emanuel_velzi/32/66975_2.png) [@Emanuel\_Velzi](https://discuss.elastic.co/u/Emanuel_Velzi)
#### Post date: [June 4, 2020, 2:54pm UTC](https://discuss.elastic.co/t/apm-labels-best-practices/234924/3 "2020-06-04T14:54:18Z")

</div>

Actually, it's very difficult to ensure that all teams use the same types for the same labels.

I need some strategy on the APM cluster side to ensure that ES does not degrade so much its performance in case of parsing errors.  
Is there something I can configure to avoid ES degrade when parsing errors occur?

`ignore_malformed` doesn't work with booleans:

```auto

PUT my_index
{
  "mappings": {
    "properties": {
      "bool_one": {
        "type": "boolean",
        "ignore_malformed": true
      }
    }
  }
}

```

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "Mapping definition for [bool_one] has unsupported parameters: [ignore_malformed : true]"
      }
    ],
...

```

> [@jalvz](#):
>
> Other alternative would be to have different indices for different services, and then configure ILM policies so that you don't end up with very small daily indices.

This could work.  
Could I merge multiple indices (one per service) to have only one per day?  
What happens if two indices have fields with equal name and different type?

---

<div class="post-metadata">

### Author: ![jalvz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jalvz/32/45613_2.png) [@jalvz](https://discuss.elastic.co/u/jalvz)
#### Post date: [June 8, 2020, 8:31am UTC](https://discuss.elastic.co/t/apm-labels-best-practices/234924/4 "2020-06-08T08:31:14Z")

</div>

Hi,

You are right that `ignore_malformed` only work with some types. Mapping errors degrade performance because throwing exceptions are expensive operations in Java, that has nothing to do with APM or APM Server. So the best thing to do is to avoid them in first place.

> it's very difficult to ensure that all teams use the same types for the same labels

Another option then is to force them to use different labels, eg. by prefixing labels names with the service name.

> Could I merge multiple indices (one per service) to have only one per day?  
> What happens if two indices have fields with equal name and different type?

There is no API or tool out of the box to merge indices, you would need to create an script to merge indices yourself.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2020, 8:33am UTC](https://discuss.elastic.co/t/apm-labels-best-practices/234924/5 "2020-07-06T08:33:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
