# Transactions tab empty but data with \`transaction.id\` available in \`apm-\*\` indices

**URL:** <https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057>\
**Category:** APM\
**Tags:** python, ui\
**Created:** [January 7, 2020, 1:07pm UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057 "2020-01-07T13:07:29Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marcin\_Kawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kawa/32/18327_2.png) [@Marcin\_Kawa](https://discuss.elastic.co/u/Marcin_Kawa)\
**Post date:** [January 7, 2020, 1:07pm UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057/1 "2020-01-07T13:07:30Z")

</div>

**Kibana version** : 7.4.0

**Elasticsearch version** : 7.4.0

**APM Server version** : 7.4.0

**APM Agent language and version** : Python / 5.3.2

**Original install method** : Docker

**Fresh install or upgraded from other version?** ELK upgraded from v6 but APM installed only after v7.4.0 has been installed

* * *

Hello,

I have just installed the APM server and APM Python (Django) client (as instructed in the documentation). The logs are streaming correctly and I can see now my code traces and detailed trace samples but I can't see any transactions even though they're collected and I can see them in the `Discover` section.

There is no relevant JS errors to prevent the app loading though if I inspect the AJAX requests I can see that `transaction_types` come back as empty list.

Any help would be greatly appreciated! (Trying to debug that issue second day 😛 )

 ![Screenshot from 2020-01-07 12-52-15](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b45f48d49b7a864111a6f9a70a1a9ce91855756b.png)  
 ![Screenshot from 2020-01-07 12-53-43](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7ee89f9d01eb749526785a4b9e20496776a72354.png)  
 ![Screenshot from 2020-01-07 12-55-28](https://us1.discourse-cdn.com/elastic/original/3X/d/c/dc21096e5ad206d4d74372ba5852fbc22fa1512c.png) ![Screenshot from 2020-01-07 12-58-29](https://us1.discourse-cdn.com/elastic/original/3X/6/e/6ec65f9be31343e2d2f3121d3cf882d942ab16c2.png) ![Screenshot from 2020-01-07 12-58-11](https://us1.discourse-cdn.com/elastic/original/3X/3/5/35f15bd4fe6a5c5284d1ef2e432a957715320969.png)

---

<div class="post-metadata">

**Author:** ![dgieselaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dgieselaar/32/46947_2.png) [@dgieselaar](https://discuss.elastic.co/u/dgieselaar)\
**Post date:** [January 7, 2020, 5:59pm UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057/2 "2020-01-07T17:59:15Z")

</div>

Hello Marcin! We do have some related issues [1, 2], but I'm not sure if the cause here is the same. Let's try to isolate the issue a bit.

Did you by any change re-configure the index names that the APM UI uses? This issue _could_ happen if for instance the metrics and error index names are configured correctly, but the transaction index name is not. See [3] for more information about these options.

Can you also run the following queries in the Kibana console (or to ES directly) and post the response here (with sensitive information being scrubbed):

```auto
GET apm-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "service.name": "Frontend"
          }
        },
        {
          "terms": {
            "processor.event": [
              "transaction"
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-24h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "types": {
      "terms": {
        "field": "transaction.type",
        "size": 100
      }
    }
  }
}

```

```auto
GET apm-*

```

[1] [https://github.com/elastic/kibana/issues/40398](https://github.com/elastic/kibana/issues/40398)  
[2] [https://github.com/elastic/kibana/issues/53571](https://github.com/elastic/kibana/issues/53571)  
[3] [https://www.elastic.co/guide/en/kibana/current/apm-settings-in-kibana.html](https://www.elastic.co/guide/en/kibana/current/apm-settings-in-kibana.html)

---

<div class="post-metadata">

**Author:** ![Marcin\_Kawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kawa/32/18327_2.png) [@Marcin\_Kawa](https://discuss.elastic.co/u/Marcin_Kawa)\
**Post date:** [January 8, 2020, 3:10pm UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057/3 "2020-01-08T15:10:49Z")

</div>

Hello, thank you for such a quick and prominent reply.

I did have a look at the links you've provided with no success.  
Result for the first query you have asked for returned 66703 results matched:

```
{
  "took" : 29,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : []
  },
  "aggregations" : {
    "types" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "request",
          "doc_count" : 66703
        }
      ]
    }
  }
}

```

The second query produced enormous output so you can find it pasted as a GitHub code snippet: [APM gist](https://gist.github.com/kawa-marcin/08fa164af973c222a67f32184047a4d9)

Additionally I have changed the config a bit to group the APM indices by month rather than day. So that's how it looks currently:

```
# apm-server.yml
#================================= Template =================================

# A template is used to set the mapping in Elasticsearch.
# By default template loading is enabled and the template is loaded.
# These settings can be adjusted to load your own template or overwrite existing ones.

# Set to false to disable template loading.
#setup.template.enabled: true

# Template name. By default the template name is "apm-%{[observer.version]}"
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.name: "apm-%{[observer.version]}"

# Template pattern. By default the template pattern is "apm-%{[observer.version]}-*" to apply to the default index settings.
# The first part is the version of apm-server and then -* is used to match all daily indices.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.pattern: "apm-%{[observer.version]}-*"

# Path to fields.yml file to generate the template.
#setup.template.fields: "${path.config}/fields.yml"

# Overwrite existing template.
setup.template.overwrite: true

#================================ Outputs =================================

# Configure the output to use when sending the data collected by apm-server.

#-------------------------- Elasticsearch output --------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (`http` and `9200`).
  # In case you specify and additional path, the scheme is required: `http://localhost:9200/path`.
  # IPv6 addresses should always be defined as: `https://[2001:db8::1]:9200`.
  hosts: ["localhost:9200"]

  # Be aware that you can only specify one Elasticsearch template.
  # If you modify the index patterns you must also update these configurations accordingly,
  # as they need to be aligned:
  # * `setup.template.name`
  # * `setup.template.pattern`
  index: "apm-%{[observer.version]}-%{+yyyy.MM}"
  indices:
   - index: "apm-%{[observer.version]}-sourcemap"
     when.contains:
       processor.event: "sourcemap"

   - index: "apm-%{[observer.version]}-error-%{+yyyy.MM}"
     when.contains:
       processor.event: "error"

   - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM}"
     when.contains:
       processor.event: "transaction"

   - index: "apm-%{[observer.version]}-span-%{+yyyy.MM}"
     when.contains:
       processor.event: "span"

   - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM}"
     when.contains:
       processor.event: "metric"

   - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM}"
     when.contains:
       processor.event: "onboarding"

```

And the indices look like:

```
curl -X GET -u undefined:$ESPASS "localhost:9200/_cat/indices/*apm*?v&s=index&pretty"

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .apm-agent-configuration h0EfO5DySGamEGDvqCXFiA 1 0 0 0 283b 283b
green open apm-7.4.0-error-2020.01 evUKYTxBTtGPVSc-0aoZFA 1 0 27 0 333.9kb 333.9kb
green open apm-7.4.0-metric-2020.01 ocvhc1wOS8KlKp5Qf5gtBg 1 0 476240 0 57mb 57mb
green open apm-7.4.0-onboarding-2020.01 qfP62XNyROCTD_gm0N9Hpg 1 0 3 0 18.3kb 18.3kb
green open apm-7.4.0-span-2020.01 4rJQLrdHRUuHQIWKv-EXKQ 1 0 11974107 0 2gb 2gb
green open apm-7.4.0-transaction-2020.01 X1JbZgCrTlmgS2cDRVBlzQ 1 0 75341 0 43.7mb 43.7mb
```

---

<div class="post-metadata">

**Author:** ![dgieselaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dgieselaar/32/46947_2.png) [@dgieselaar](https://discuss.elastic.co/u/dgieselaar)\
**Post date:** [January 12, 2020, 7:54pm UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057/4 "2020-01-12T19:54:01Z")

</div>

> [@Marcin\_Kawa](#):
>
> 119

That all looks properly configured. Your mapping is OK and you can aggregate on transaction.type. I'm having a hard time coming up with a reason why this happens. You didn't change any configuration in Kibana I assume? (kibana.yml rather than apm-server.yml).

---

<div class="post-metadata">

**Author:** ![Marcin\_Kawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kawa/32/18327_2.png) [@Marcin\_Kawa](https://discuss.elastic.co/u/Marcin_Kawa)\
**Post date:** [January 14, 2020, 1:40pm UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057/5 "2020-01-14T13:40:15Z")

</div>

Hello Dario, I have not modified the `kibana.yml` config file though I still haven't gave up on that and trying to find the solution for the issue. Is this the part of the code responsible for retrieving the transaction data [get\_service\_transaction\_types.ts#L21](https://github.com/elastic/kibana/blob/06bee6051258009cca6c0c92fa1362c56823e734/x-pack/legacy/plugins/apm/server/lib/services/get_service_transaction_types.ts#L21)?

I don't have the `apm_oss.transactionIndices` in my Elasticsearch. Could that be the issue? If not could you please pint me where are the endpoints for the APM implemented? I'm happy to debug this to find some solution.

Cheers!

---

<div class="post-metadata">

**Author:** ![Marcin\_Kawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kawa/32/18327_2.png) [@Marcin\_Kawa](https://discuss.elastic.co/u/Marcin_Kawa)\
**Post date:** [January 14, 2020, 1:54pm UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057/6 "2020-01-14T13:54:49Z")

</div>

Hum, I understand now, the `apm_oss.transactionIndices` is mapped to the `apm-*` by default so this can't be the problem. Could you point me to the endpoints implementation anyway? I'll double check what is returned by the endpoints and why.

---

<div class="post-metadata">

**Author:** ![Marcin\_Kawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kawa/32/18327_2.png) [@Marcin\_Kawa](https://discuss.elastic.co/u/Marcin_Kawa)\
**Post date:** [January 14, 2020, 3:23pm UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057/7 "2020-01-14T15:23:50Z")

</div>

Ufffff, solved it! It was partially my fault of not following the rule of "Never name your staff with space"! So the full APM service name was originally with space character e.g: `Frontend app` (I have stripped it for the examples here).

By following the logic in your [apm/server/routes/services.ts#L39](https://github.com/elastic/kibana/blob/v7.4.0/x-pack/legacy/plugins/apm/server/routes/services.ts#L39) I have found out that the service name is not properly URL decoded when sending to the view function for the endpoins (e.g: `Frontend app` will be sent as `Frontend%20app`) hence the results would come back empty.

# Solution: Don't name your services with space character in the middle!

Woohoo! Most important it works now! Thanks a lot for the support. It's probably worth pointing this out in the documentation or fixing the URL decoding.

Hope it will help somebody! I took me way to long to figure this out!

Cheers!

---

<div class="post-metadata">

**Author:** ![dgieselaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dgieselaar/32/46947_2.png) [@dgieselaar](https://discuss.elastic.co/u/dgieselaar)\
**Post date:** [January 15, 2020, 8:32am UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057/8 "2020-01-15T08:32:35Z")

</div>

Nice research! This is a bug on our side, because the docs [1] say that we support spaces in service.name:

> The service name can only contain alphanumeric characters, spaces, underscores, and dashes (must match `^[a-zA-Z0-9 _-]+$` ).

We'll look into it. I've filed an issue if you want to track it [2].

[1] [https://www.elastic.co/guide/en/apm/get-started/current/agents.html](https://www.elastic.co/guide/en/apm/get-started/current/agents.html)  
[2] [[APM] Spaces in service.name are not properly being decoded · Issue #54884 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/54884)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2020, 4:32am UTC](https://discuss.elastic.co/t/transactions-tab-empty-but-data-with-transaction-id-available-in-apm-indices/214057/9 "2020-02-05T04:32:38Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.