Uploading sourcemaps from AIX webserver

Hello,

I can't figure out how to upload sourcemaps to kibana.

**Config for APM**
// using full condition instead of PROD variable (webpack will remove this code if syntax is like this)
if (process.env.NODE_ENV === 'production') {
  window.amp = initApm({
    // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
    serviceName: `web-fe - ${window.location.hostname}`.replaceAll('.', '-'),
    // Set custom APM Server URL (default: http://localhost:8200)
    serverUrl: `${window.location.origin}/apm/`, // 'https://dweb.test.lab:8443/apm/',
    // Set service version (required for sourcemap feature)
    serviceVersion: '', //process.env.WEB_PORTAL_VERSION,
    logLevel: 'debug'
  });
}

in apm-server I see following logs

{"log.level":"info","@timestamp":"2022-04-28T16:31:35.948+0200","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/rum/events","http.request.method":"POST","user_agent.original":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0","source.address":"10.10.241.246","http.request.body.bytes":1904,"http.request.id":"e2f98125-8b5f-4e1c-9d05-3b48016db680","event.duration":274665,"http.response.status_code":202,"ecs.version":"1.6.0"}'

I also dont understand documentation

How does it work after sourcemaps are created ? Do I need to upload them just to apm-server and will it automatically upload it to kibana ?

Our webserver is running on AIX so uploading via curl is not possible (It is forbidden to install packages like curl on it)

With new release of app, sourcemaps are generated and I can find them .chunk.js.map files on webserver, but I cant figure it out how to get to the point that I would see stacktrace in kibana

Configuration on Elasticsearch and apm-server is correct, version compatibility for elk stack is also correct for sourcemaps to work, kibana endpoint in apm-server is configured, I have verified it on this forum some time ago.

Hi @Hamunaptroid,

Thank you for contacting us!

The snippet you shared with us corresponds to the agent initialisation and it's not related to the logic for uploading a sourcemap.

The log that you are seeing in apm-server means that the RUM agent sent events properly: /intake/v2/rum/events is the endpoint where the agent sends events.

When it comes to the sourcemap, you are right, you need to upload them to Kibana. Recently, we updated one of the tests that checks the sourcemap upload logic works.

You will see there that we are not using curl and instead we rely on the library request (but you can use a different one if you want) to upload the sourcemap. Perhaps you can add part of this logic as an additional step of your release pipeline.

Let me know if this helps you.

Thanks,
Alberto

Hello Alberto,

thank you for the reply,
I forgot to share version of apm and kibana - both are v7.17.0

so if I understand it correctly, sourcemaps should be uploaded only to kibana and not to apm-server ? In 3rd step on following documentation, it says
"Enable and upload the source map to APM Server"

My goal is for sourcemaps to be automatically uploaded via "custom app" during build/deployment of new version of the app. So it means that I should include "uploading via custom app" in our build/deployment pipeline right ?

Hi @Hamunaptroid,

If you are using the 7.17.0 version then you need to upload the sourcemap to apm-server. The link you shared with us points to the 8.1 version.

The documentation for 7.17 is the following one:

You are right, including it in your build/deployment is a good strategy. The code snippet in the documentation is a good one, but in case it helps you, this link shows you a test that checks the sourcemap upload logic for versions older than 8.x.

Hope this helps

Thanks,
Alberto

Hello @Alberto_Delgado ,

thank you for the explanation.
Right now I am trying to manually upload sourcemaps to apm server via curl just to test if it will really work, but even after I upload some of our sourcemaps, I still can't see minified code.

After Elastic has indexed sourcemaps I was able to get some "hits" (i copied only part of it)

{
  "took" : 506,
  "timed_out" : false,
  "_shards" : {
    "total" : 246,
    "successful" : 246,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 12,
      "relation" : "eq"
    },
    "max_score" : 0.03922071,
    "hits" : [
      {
        "_index" : "apm-7.17.0-sourcemap",
        "_type" : "_doc",
        "_id" : "l786moABNwLnXlT6qekM",
        "_score" : 0.03922071,
        "_source" : {
          "observer" : {
            "hostname" : "mvmperfls1",
            "id" : "c9ef030a-e37d-4442-9475-b1d0691fb8db",
            "type" : "apm-server",
            "ephemeral_id" : "043e0a6e-599e-4795-9075-042324062ecb",
            "version" : "7.17.0",
            "version_major" : 7
          },
          "@timestamp" : "2022-05-06T16:36:16.066Z",
          "ecs" : {
            "version" : "1.12.0"
          },
          "sourcemap" : {
            "bundle_filepath" :

....... etc

When I opened any error index in kibana I didn't see any minified code it just points on the specific chunk.js files.... For example, I have uploaded sourcemap named "505.007504a0" and then searched for error that contains chunk "505.007504a0". I also checked in Kibana if the sourcemap 505.007 was indexed.

... but it is still not working ?

Any ideas ?

Hi @Hamunaptroid,

Apologies for the long delay! We are still checking this and what I have seen is that if you update the elastic stack to version 8.x.x it should work properly. Is that update something that would be feasible for you?

One question: are you seeing the sourcemaps listed when accessing http://localhost:5601/api/apm/sourcemaps? (the url corresponds to Kibana, bear in mind that perhaps your Kibana domain is different)

Thanks,
Alberto

Hello @Alberto_Delgado,

Thank you,

I can't see anything when I open the link (with my kibana of course). So what does it exactly mean ? Sourcemaps are indexed by Elasticsearch but they are not accessible by kibana ?
I don't understand why I was able to find uploaded sourcemps in Kibana as I showed, but can't see anything on /api/apm/sourcemaps.

We will upgrade ELK stack to version 8.2

Hello @Alberto_Delgado,

I am having 2nd thoughts about upgrading our ELK stack to 8.2
I am worried about necessity of using fleet which requires security features enabled.

I am not sure if apm-server is entirely replaced by fleet in 8.2. ...
After apm integration the last step in this documentation is to stop legacy apm server process

but in step 5. it says at the end to start apm-server (after apmm integration)

So apm-server is no longer needed and can be used in 8.2 only to write to "well-defined data streams" (whatever that means). In future releases is apm-server going to be depreciated and removed?

You wrote that if I update elastic to 8.2 it should work. Why it should work in new version? Some time ago, we upgraded elk stack from older versions, I think 6.x to 7.x because of sourcemaps and it did nothing. Were you able to find what exactly is wrong with it now ?

Thank you

Hi @Hamunaptroid ,
there is no known issue with sourcemaps on 7.17 for standalone APM Server. So it is not required to upgrade to 8.x for using sourcemaps.

As my colleague already pointed out, in 7.17 for standalone APM Server you need to upload the sourcemap via APM Server (not via Kibana). Can you please try again by following these steps to find out where the issue lies:

(0) Start up APM Server with logging level set to debug.
(1) Upload the sourcemap via APM Server and then query it by service.name, service.version and bundle_filepath from Kibana. These three attributes form a unique key. There should only be 1 sourcemap with this key. If you already uploaded multiple of them, please delete the redundant ones.
(2) Ingest an example document matching the service.name, service.version and bundle_filepath (path in the documents stack trace).
For testing purposes, you can reuse a test document , but you need to set the service.name, service.version and path according to the values of your uploaded sourcemap.
You can upload the document for example via curl

curl -i -k -H "Content-type: application/x-ndjson" -H "Authorization: Bearer ${ELASTIC_APM_SECRET_TOKEN}" --data-binary @example.ndjson ${ELASTIC_APM_SERVER_URL}/intake/v2/rum/events

(3) Fetch the specific test document from Elasticsearch and check if sourcemapping has been applied. If so, it shows that there is no general issue, but most certainly a problem with your sourcemap or your data. In this case, please double check that the mentioned keys (service.name, service.version and bundle_filepath==path) are identical in your sourcemap and the ingested error events.

If sourcemapping has not been applied, then please share the apm-server logs, the metadata of your sourcemap and the example document that you uploaded with us and we can take a closer look (remove any sensitive information before sharing).