Transactions tab empty but data with `transaction.id` available in `apm-*` indices

[Marcin_Kawa]

Kibana version: 7.4.0

Elasticsearch version: 7.4.0

APM Server version: 7.4.0

APM Agent language and version: Python / 5.3.2

Original install method: Docker

Fresh install or upgraded from other version? ELK upgraded from v6 but APM installed only after v7.4.0 has been installed

---

Hello,

I have just installed the APM server and APM Python (Django) client (as instructed in the documentation). The logs are streaming correctly and I can see now my code traces and detailed trace samples but I can't see any transactions even though they're collected and I can see them in the Discover section.

There is no relevant JS errors to prevent the app loading though if I inspect the AJAX requests I can see that transaction_types come back as empty list.

Any help would be greatly appreciated! (Trying to debug that issue second day )

[dgieselaar (Dario Gieselaar)]

Hello Marcin! We do have some related issues [1, 2], but I'm not sure if the cause here is the same. Let's try to isolate the issue a bit.

Did you by any change re-configure the index names that the APM UI uses? This issue could happen if for instance the metrics and error index names are configured correctly, but the transaction index name is not. See [3] for more information about these options.

Can you also run the following queries in the Kibana console (or to ES directly) and post the response here (with sensitive information being scrubbed):

GET apm-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "service.name": "Frontend"
          }
        },
        {
          "terms": {
            "processor.event": [
              "transaction"
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-24h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "types": {
      "terms": {
        "field": "transaction.type",
        "size": 100
      }
    }
  }
}

GET apm-*

[1] https://github.com/elastic/kibana/issues/40398
[2] https://github.com/elastic/kibana/issues/53571
[3] https://www.elastic.co/guide/en/kibana/current/apm-settings-in-kibana.html

[Marcin_Kawa]

Hello, thank you for such a quick and prominent reply.

I did have a look at the links you've provided with no success.
Result for the first query you have asked for returned 66703 results matched:

{
  "took" : 29,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [  ]
  },
  "aggregations" : {
    "types" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "request",
          "doc_count" : 66703
        }
      ]
    }
  }
}

The second query produced enormous output so you can find it pasted as a GitHub code snippet: APM gist

Additionally I have changed the config a bit to group the APM indices by month rather than day. So that's how it looks currently:

# apm-server.yml
#================================= Template =================================

# A template is used to set the mapping in Elasticsearch.
# By default template loading is enabled and the template is loaded.
# These settings can be adjusted to load your own template or overwrite existing ones.

# Set to false to disable template loading.
#setup.template.enabled: true

# Template name. By default the template name is "apm-%{[observer.version]}"
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.name: "apm-%{[observer.version]}"

# Template pattern. By default the template pattern is "apm-%{[observer.version]}-*" to apply to the default index settings.
# The first part is the version of apm-server and then -* is used to match all daily indices.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.pattern: "apm-%{[observer.version]}-*"

# Path to fields.yml file to generate the template.
#setup.template.fields: "${path.config}/fields.yml"

# Overwrite existing template.
setup.template.overwrite: true

#================================ Outputs =================================

# Configure the output to use when sending the data collected by apm-server.

#-------------------------- Elasticsearch output --------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (`http` and `9200`).
  # In case you specify and additional path, the scheme is required: `http://localhost:9200/path`.
  # IPv6 addresses should always be defined as: `https://[2001:db8::1]:9200`.
  hosts: ["localhost:9200"]

  # Be aware that you can only specify one Elasticsearch template.
  # If you modify the index patterns you must also update these configurations accordingly,
  # as they need to be aligned:
  # * `setup.template.name`
  # * `setup.template.pattern`
  index: "apm-%{[observer.version]}-%{+yyyy.MM}"
  indices:
   - index: "apm-%{[observer.version]}-sourcemap"
     when.contains:
       processor.event: "sourcemap"

   - index: "apm-%{[observer.version]}-error-%{+yyyy.MM}"
     when.contains:
       processor.event: "error"

   - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM}"
     when.contains:
       processor.event: "transaction"

   - index: "apm-%{[observer.version]}-span-%{+yyyy.MM}"
     when.contains:
       processor.event: "span"

   - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM}"
     when.contains:
       processor.event: "metric"

   - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM}"
     when.contains:
       processor.event: "onboarding"

And the indices look like:

curl -X GET -u undefined:$ESPASS "localhost:9200/_cat/indices/*apm*?v&s=index&pretty"

health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .apm-agent-configuration      h0EfO5DySGamEGDvqCXFiA   1   0          0            0       283b           283b
green  open   apm-7.4.0-error-2020.01       evUKYTxBTtGPVSc-0aoZFA   1   0         27            0    333.9kb        333.9kb
green  open   apm-7.4.0-metric-2020.01      ocvhc1wOS8KlKp5Qf5gtBg   1   0     476240            0       57mb           57mb
green  open   apm-7.4.0-onboarding-2020.01  qfP62XNyROCTD_gm0N9Hpg   1   0          3            0     18.3kb         18.3kb
green  open   apm-7.4.0-span-2020.01        4rJQLrdHRUuHQIWKv-EXKQ   1   0   11974107            0        2gb            2gb
green  open   apm-7.4.0-transaction-2020.01 X1JbZgCrTlmgS2cDRVBlzQ   1   0      75341            0     43.7mb         43.7mb

[dgieselaar (Dario Gieselaar)]

@Marcin_Kawa

119

That all looks properly configured. Your mapping is OK and you can aggregate on transaction.type. I'm having a hard time coming up with a reason why this happens. You didn't change any configuration in Kibana I assume? (kibana.yml rather than apm-server.yml).

[Marcin_Kawa]

Hello Dario, I have not modified the kibana.yml config file though I still haven't gave up on that and trying to find the solution for the issue. Is this the part of the code responsible for retrieving the transaction data get_service_transaction_types.ts#L21?

I don't have the apm_oss.transactionIndices in my Elasticsearch. Could that be the issue? If not could you please pint me where are the endpoints for the APM implemented? I'm happy to debug this to find some solution.

Cheers!

[Marcin_Kawa]

Hum, I understand now, the apm_oss.transactionIndices is mapped to the apm-* by default so this can't be the problem. Could you point me to the endpoints implementation anyway? I'll double check what is returned by the endpoints and why.

[Marcin_Kawa]

Ufffff, solved it! It was partially my fault of not following the rule of "Never name your staff with space"! So the full APM service name was originally with space character e.g: Frontend app (I have stripped it for the examples here).

By following the logic in your apm/server/routes/services.ts#L39 I have found out that the service name is not properly URL decoded when sending to the view function for the endpoins (e.g: Frontend app will be sent as Frontend%20app) hence the results would come back empty.

Solution: Don't name your services with space character in the middle!

Woohoo! Most important it works now! Thanks a lot for the support. It's probably worth pointing this out in the documentation or fixing the URL decoding.

Hope it will help somebody! I took me way to long to figure this out!

Cheers!

[dgieselaar (Dario Gieselaar)]

Nice research! This is a bug on our side, because the docs [1] say that we support spaces in service.name:

The service name can only contain alphanumeric characters, spaces, underscores, and dashes (must match ^[a-zA-Z0-9 _-]+$ ).

We'll look into it. I've filed an issue if you want to track it [2].

[1] https://www.elastic.co/guide/en/apm/get-started/current/agents.html
[2] [APM] Spaces in service.name are not properly being decoded · Issue #54884 · elastic/kibana · GitHub