Sourcemaps are not getting applied

Elastic Observability APM
1

Hello, I have been having problems with RUM agent. Stack traces are not shown. In Kibana there is just "stack trace is not available".

In APM server I can see this error:

2020-08-11T09:35:34.035+0200 WARN [stacktrace] model/stacktrace.go:62 Cannot apply sourcemap without a service name or service version

but it gets accepted ...

2020-08-11T11:04:36.599+0200 INFO [request] middleware/log_middleware.go:97 request accepted {"request_id": "52d735b4-45a2-4252-b7b8-a5e3a5308184", "method": "POST", "URL": "/intake/v2/rum/events", "content_length": 19119, "remote_address": " 10.10.22.14 ", "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)", "response_code": 202}

This is what apm-server.yml looks like:

rum:

enabled: true

apm-server.rum.enabled: true

apm-server.rum.source_mapping.enabled: true

apm-server.rum.allow_origins: ['*']

apm-server.rum.source_mapping.index_pattern: "apm-*"

And this is .js that we use to configure sourcemaps:

window.REACT_APP_AMP_CONFIG_ENABLED = true;

window.REACT_APP_AMP_CONFIG = {

// Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)

serviceName: 'FrontEnd',

// Set custom APM Server URL (default: http://localhost:8200)

serverUrl: 'https://FrontEnd.pls.lab:8443/apm/',

// Set service version (required for sourcemap feature)

serviceVersion: '',

//logLevel: 'debug'

I have played with apm-config a little, I have tried to put something into serviceVersion: ... but nothing happens, chunk.js files didn't have full path in "//# sourceMappingURL=" so I added full path to the sourceMaps ... its still the same and still same error that It cant apply sourcemaps, so in Kibana its just "no stack trace available"

I am lost ... Nothing I tried helps. I would be grateful for any suggestions.

Thank you.

Whole elastic stack is on 7.8

2

Hello,

Just to make sure because you don't explicitly mention it: Sourcemaps need to be uploaded to APM Server first. Did you follow instructions in https://www.elastic.co/guide/en/apm/server/current/sourcemap-api.html?

If so, what result are you seeing from executing the request?
You can also search the apm-sourcemap index in Elasticserch: do you get any hits?

3

Hello,

Thank you for reply,

Sourcemaps should be getting uploaded via custom app.
I have searched apm-sourcemap index and this is what I see:

GET apm-*/_search
{
  "query": {
    "term": {
      "processor.event": "sourcemap"
    }
  }
}

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 172,
    "successful" : 172,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [  ]
  }
}

so does it mean sourcemaps are not really in our apm-server ?
Can be serviceVersion empty ? serviceName is set.

Thank you.

4

Hello!

You found it, serviceVersion can't be empty, so your upload request is failing (presumably silently).
You need to upload your sourcemaps with a serviceVersion matching the one you use for configuring the agent, see: https://www.elastic.co/guide/en/apm/agent/rum-js/current/sourcemap.html and https://www.elastic.co/guide/en/apm/agent/rum-js/current/configuration.html#service-version.

Normally setting the version is optional, but when using sourcemaps it is important to version the software so we can know which sourcemap to use for each release.

So pass in a version (it can be Semver, a Git hash, whatever), use the same one when initializing the agent, and run that query against Elasticsearch again to confirm it was uploaded.

Hope this helps!

1 Like
5

Hello,

Interesting ... When I check Kibana for some errors and I open it's metadata, I can see all that is needed but I still see the error message in apm logs and without stack trace.

###### Service

|**service.language.name**|javascript|
| --- | --- |
|**service.name**|FrontEnd|
|**service.version**|1.2.3|

This is just info about the agent configuration ?

Sometimes I get confused about output from my GET requests, does my request for getting sourcemaps mean that they are not in apm-server as value for "hits": is 0 ?

Thank you for advice Juan.

6

So the way it works is that you use the APM Server Sourcemap API linked above to POST sourcemaps to APMServer. APM Server will validate them and index the in Elasticsearch.
So yes, if you GET request to search sourcemaps in Elasticsearch returns 0 hits, it means that something went wrong and nothing was uploaded. That might be because e.g. a connection problem between APM Server and Elasticsearch, or a validation failure (like no service version given).

Once you have a correct sourcemap uploaded, errors coming from then on will have readable stack traces. But the fix is not retroactive: you won't be able to see stack traces from errors ingested before the sourcemap has been uploaded.

I hope that makes sense.

7

Hello Juan,

You are the absolute King.
Error "cannot apply sourcemaps" dissappeared.

I have also found out that our source maps don't have mappings, therefore they can't be uploaded to apm-server. But this is not elastic issue, Long story short - it is issue during generation sourcemaps. (with wrong JS version as I was told by our devs).

I was looking for a root cause for this for a pretty long time, so your advice was extremely helpful.

Thank you very much.

8

Hello,

Happy to hear you figured it out. Let me know if new issues arise!

9

Hello again Juan,
It seems like issue is not fixed after all :frowning:

I want to check if sourcemaps were correctly uploaded this time (after I manually uploaded them and got no errors during uploading). When I try to check them in in Kibana, I get error
"Too much recursion"

GET apm-*/_search
{
  "query": {
    "term": {
      "processor.event": "sourcemaps"
    }
  }
}

Errors look like this:
Dev Tools Console output
Unknown Request Error
too much recursion

expandLiteralStrings@http://muhServer.pls.work:5601/31997/bundles/plugin/esUiShared/esUiShared.plugin.js:1:794297
EditorOutputUI/<@http://muhServer.pls.work:5601/31997/bundles/plugin/console/2.plugin.js:1:393171
fs@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:77781
Ml@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:102513
__kbnSharedDeps__</t.unstable_runWithPriority@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:350:3462
Hi@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:45442
wl@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:102276
ol@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:87214
Gi/<@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:45733
__kbnSharedDeps__</t.unstable_runWithPriority@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:350:3462
Hi@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:45442
Gi@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:45680
Yi@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:45613
el@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:84080
Aa@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:342:62991
_callee$@http://muhServer.pls.work:5601/31997/bundles/plugin/console/2.plugin.js:1:256432
l@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288:969217
s/o._invoke</<@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288:968971
_/</e[t]@http://muhServer.pls.work:5601/31997/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288:969574
asyncGeneratorStep@http://muhServer.pls.work:5601/31997/bundles/plugin/console/2.plugin.js:1:254090
_next@http://muhServer.pls.work:5601/31997/bundles/plugin/console/2.plugin.js:1:254419

in addition, new indices are now being named only as (for example) "apm-7.8.0-2020-26.08" instead of them having some specification what exactly it is ... (error, transaction, span ...).

...

Have you ever seen anything like this?

Thank you.

10

Hello,

As for the crash: you are hitting https://github.com/elastic/kibana/pull/72628, that will be fixed in 7.9.1 I believe. In the meanwhile, go to the settings tab in the console and disable the option saying "Use triple quotes in output pane".

As for the index names. Can you recall what changed since the indices had the right names? Did you redeploy apm-server, updated apm-server or agent version, etc?
Running apm-server -e setup template should hopefully get it back on track.

Hope that helps!

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.