APM viewer in kibana broken when transaction name has a slash in it

sgaron_cse · April 9, 2019, 2:21pm

Kibana version: 6.7.1

Elasticsearch version: 6.7.1

APM Server version: 6.7.1

APM Agent language and version: Python 4.2.1 [Flask integration]

Browser version: Chrome 73.0.3683.103

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Official docker containers

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? No special config

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

When using the flask integration of ElasticAPM, metrics are collected correctly into the elasticsearch cluster but when trying to view the data, there is a bug in how kibana calls its own APM APIs because of the slashes the ElasticAPM put in the transaction name.

For example, viewing the transactions for "GET /index.html" makes the following two API calls return a 404 not found:

It looks like the ~20 and the ~2F for the original kibana url (http://localhost:5601/app/apm#/my_app/transactions/request/GET~20~2Findex.html) are translated to %20 and %2F which confuses the API endpoint. If you replace the % in the failing urls with ~ they return the right results.

Steps to reproduce:

Gather ElasticAPM metrics from any flask app
Click on the APM metrics in kibana for a given page
No transaction will be found and you will get a 404 for the chart and distribution APM APIs

Errors in browser console (if relevant):

GET http://localhost:5601/api/apm/services/my_app/transaction_groups/request/GET%20%2Findex.html/distribution?start=2019-04-09T12%3A45%3A10.735Z&end=2019-04-09T13%3A45%3A10.736Z 404 (Not Found)
(anonymous)                     @ commons.bundle.js:3
Promise.then (async)
(anonymous)                     @ commons.bundle.js:3
step                            @ vendors.bundle.dll.js:531
(anonymous)                     @ vendors.bundle.dll.js:531
(anonymous)                     @ vendors.bundle.dll.js:531
__awaiter                       @ vendors.bundle.dll.js:531
kfetch                          @ commons.bundle.js:3
_callee$                        @ apm.bundle.js:2
tryCatch                        @ vendors.bundle.dll.js:497
invoke                          @ vendors.bundle.dll.js:497
prototype.(anonymous function)  @ vendors.bundle.dll.js:497
step                            @ apm.bundle.js:2
(anonymous)                     @ apm.bundle.js:2
(anonymous)                     @ apm.bundle.js:2
callApi                         @ apm.bundle.js:2
_callee2$                       @ apm.bundle.js:2
tryCatch                        @ vendors.bundle.dll.js:497
invoke                          @ vendors.bundle.dll.js:497
prototype.(anonymous function)  @ vendors.bundle.dll.js:497
step                            @ apm.bundle.js:2
(anonymous)                     @ apm.bundle.js:2
Promise.then (async)
step                            @ apm.bundle.js:2
(anonymous)                     @ apm.bundle.js:2
(anonymous)                     @ apm.bundle.js:2
loadTransactionDistribution     @ apm.bundle.js:2
_callee$                        @ vendors.bundle.dll.js:491
tryCatch                        @ vendors.bundle.dll.js:497
invoke                          @ vendors.bundle.dll.js:497
prototype.(anonymous function)  @ vendors.bundle.dll.js:497
step                            @ vendors.bundle.dll.js:491
(anonymous)                     @ vendors.bundle.dll.js:491
(anonymous)                     @ vendors.bundle.dll.js:491
maybeFetchData                  @ vendors.bundle.dll.js:491
_callee3$                       @ vendors.bundle.dll.js:491
tryCatch                        @ vendors.bundle.dll.js:497
invoke                          @ vendors.bundle.dll.js:497
prototype.(anonymous function)  @ vendors.bundle.dll.js:497
step                            @ vendors.bundle.dll.js:491
(anonymous)                     @ vendors.bundle.dll.js:491
(anonymous)                     @ vendors.bundle.dll.js:491
componentDidUpdate              @ vendors.bundle.dll.js:491
Hi                              @ vendors.bundle.dll.js:212
Fi                              @ vendors.bundle.dll.js:212
Gi                              @ vendors.bundle.dll.js:212
ii                              @ vendors.bundle.dll.js:212
xg                              @ vendors.bundle.dll.js:212
enqueueSetState                 @ vendors.bundle.dll.js:212
G.setState                      @ vendors.bundle.dll.js:180
onStateChange                   @ vendors.bundle.dll.js:149
dispatch                        @ vendors.bundle.dll.js:76
(anonymous)                     @ vendors.bundle.dll.js:497
(anonymous)                     @ apm.bundle.js:2
setInterval (async)
updateRefreshRate               @ apm.bundle.js:2
(anonymous)                     @ apm.bundle.js:2
listener                        @ vendors.bundle.dll.js:62
(anonymous)                     @ vendors.bundle.dll.js:62
notifyListeners                 @ vendors.bundle.dll.js:62
setState                        @ vendors.bundle.dll.js:111
(anonymous)                     @ vendors.bundle.dll.js:111
confirmTransitionTo             @ vendors.bundle.dll.js:62
handlePop                       @ vendors.bundle.dll.js:111
handleHashChange                @ vendors.bundle.dll.js:111

Provide logs and/or server output (if relevant):
No error server side...

Workaround:
For now, if I want to see all the the transactions properly I have to monkey patch the build_name_with_http_method_prefix from the ElasticAPM utils library to strip all slashes.

def monkey_build_name_with_http_method_prefix(name, request):
    name = name.strip("/").replace("/", ".")
    return " ".join((request.method, name)) if name else "none"
   
from elasticapm import utils
utils.build_name_with_http_method_prefix = monkey_build_name_with_http_method_prefix

This is obviously not the proper way to fix this. Kibana APM APIs should be able to handle request with slashes in the name.

sqren · April 9, 2019, 10:54pm

Hi there,

Thanks for reaching out. I agree, forward slashes in transaction names should definitely be supported. Unfortunately I have not been able reproduce you issue, so I will need a little more info from you.

From the URLs you have posted, everything looks right. Kibana uses a custom encoding scheme which uses ~ (tilde) instead of % (percent), and the API requests to the backend uses normal encoding.

Depending on context you should therefore see three different representations of the transaction name:
decoded: GET /index.html
normal encoding: GET%20%2Findex.html
custom encoding: GET~20~2Findex.html

I'm therefore quite curious as to why the API requests return 404. I would expect the following curl request to return 200:

curl -v http://localhost:5601/api/apm/services/my_app/transaction_groups/request/GET%20%2Findex.html/distribution?start=2019-04-09T12%3A45%3A10.735Z&end=2019-04-09T13%3A45%3A10.736Z

I don't suppose you have any middleware or proxy when you call http://localhost:5601?

sgaron_cse · April 10, 2019, 1:07am

O_o

You totally nailed it! That was nginx fault...

Just in case someone stumble on this thread:
If you add a / at the end of the proxy_pass directive in nginx, links are decode before being passed to the backend server. I removed the slash and now all is good.

proxy_pass http://kibana:5601; --> GOOD

proxy_pass http://kibana:5601/; --> BAD

Thanks a lot for the help and sorry that you had to test this.

Steve

sqren · April 10, 2019, 12:35pm

Hi Steve,

Glad you found a workaround. And thanks for sharing it with others!
I'll look into what I can do to mitigate this on our end.

sgaron_cse · April 10, 2019, 7:09pm

To be honest, I don't think there is much you can do from your end. This was mainly my fault.

I guess maybe the only thing could be to have the APIs use ~ instead of % but even then that's just fixing a problem that the user's introduced themselves...

system · May 1, 2019, 3:09pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.