Uploading sourcemaps from AIX webserver

[Hamunaptroid]

Hello,

I can't figure out how to upload sourcemaps to kibana.

**Config for APM**
// using full condition instead of PROD variable (webpack will remove this code if syntax is like this)
if (process.env.NODE_ENV === 'production') {
  window.amp = initApm({
    // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
    serviceName: `web-fe - ${window.location.hostname}`.replaceAll('.', '-'),
    // Set custom APM Server URL (default: http://localhost:8200)
    serverUrl: `${window.location.origin}/apm/`, // 'https://dweb.test.lab:8443/apm/',
    // Set service version (required for sourcemap feature)
    serviceVersion: '', //process.env.WEB_PORTAL_VERSION,
    logLevel: 'debug'
  });
}

in apm-server I see following logs

{"log.level":"info","@timestamp":"2022-04-28T16:31:35.948+0200","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/rum/events","http.request.method":"POST","user_agent.original":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0","source.address":"10.10.241.246","http.request.body.bytes":1904,"http.request.id":"e2f98125-8b5f-4e1c-9d05-3b48016db680","event.duration":274665,"http.response.status_code":202,"ecs.version":"1.6.0"}'

I also dont understand documentation

RUM source map API | Kibana Guide [8.1] | Elastic

How to apply source maps to error stack traces when using minified bundles |...

How does it work after sourcemaps are created ? Do I need to upload them just to apm-server and will it automatically upload it to kibana ?

Our webserver is running on AIX so uploading via curl is not possible (It is forbidden to install packages like curl on it)

With new release of app, sourcemaps are generated and I can find them .chunk.js.map files on webserver, but I cant figure it out how to get to the point that I would see stacktrace in kibana

Configuration on Elasticsearch and apm-server is correct, version compatibility for elk stack is also correct for sourcemaps to work, kibana endpoint in apm-server is configured, I have verified it on this forum some time ago.

[Alberto_Delgado]

Hi @Hamunaptroid,

Thank you for contacting us!

The snippet you shared with us corresponds to the agent initialisation and it's not related to the logic for uploading a sourcemap.

The log that you are seeing in apm-server means that the RUM agent sent events properly: /intake/v2/rum/events is the endpoint where the agent sends events.

When it comes to the sourcemap, you are right, you need to upload them to Kibana. Recently, we updated one of the tests that checks the sourcemap upload logic works.

You will see there that we are not using curl and instead we rely on the library request (but you can use a different one if you want) to upload the sourcemap. Perhaps you can add part of this logic as an additional step of your release pipeline.

Let me know if this helps you.

Thanks,
Alberto

[Hamunaptroid]

Hello Alberto,

thank you for the reply,
I forgot to share version of apm and kibana - both are v7.17.0

so if I understand it correctly, sourcemaps should be uploaded only to kibana and not to apm-server ? In 3rd step on following documentation, it says
"Enable and upload the source map to APM Server"

My goal is for sourcemaps to be automatically uploaded via "custom app" during build/deployment of new version of the app. So it means that I should include "uploading via custom app" in our build/deployment pipeline right ?

How to apply source maps to error stack traces when using minified bundles |...

[Alberto_Delgado]

Hi @Hamunaptroid,

If you are using the 7.17.0 version then you need to upload the sourcemap to apm-server. The link you shared with us points to the 8.1 version.

The documentation for 7.17 is the following one:

How to apply source maps to error stack traces when using minified bundles |...

You are right, including it in your build/deployment is a good strategy. The code snippet in the documentation is a good one, but in case it helps you, this link shows you a test that checks the sourcemap upload logic for versions older than 8.x.

Hope this helps

Thanks,
Alberto

[Hamunaptroid]

Hello @Alberto_Delgado ,

thank you for the explanation.
Right now I am trying to manually upload sourcemaps to apm server via curl just to test if it will really work, but even after I upload some of our sourcemaps, I still can't see minified code.

After Elastic has indexed sourcemaps I was able to get some "hits" (i copied only part of it)

{
  "took" : 506,
  "timed_out" : false,
  "_shards" : {
    "total" : 246,
    "successful" : 246,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 12,
      "relation" : "eq"
    },
    "max_score" : 0.03922071,
    "hits" : [
      {
        "_index" : "apm-7.17.0-sourcemap",
        "_type" : "_doc",
        "_id" : "l786moABNwLnXlT6qekM",
        "_score" : 0.03922071,
        "_source" : {
          "observer" : {
            "hostname" : "mvmperfls1",
            "id" : "c9ef030a-e37d-4442-9475-b1d0691fb8db",
            "type" : "apm-server",
            "ephemeral_id" : "043e0a6e-599e-4795-9075-042324062ecb",
            "version" : "7.17.0",
            "version_major" : 7
          },
          "@timestamp" : "2022-05-06T16:36:16.066Z",
          "ecs" : {
            "version" : "1.12.0"
          },
          "sourcemap" : {
            "bundle_filepath" :

....... etc

When I opened any error index in kibana I didn't see any minified code it just points on the specific chunk.js files.... For example, I have uploaded sourcemap named "505.007504a0" and then searched for error that contains chunk "505.007504a0". I also checked in Kibana if the sourcemap 505.007 was indexed.

... but it is still not working ?

Any ideas ?

[Alberto_Delgado]

Hi @Hamunaptroid,

Apologies for the long delay! We are still checking this and what I have seen is that if you update the elastic stack to version 8.x.x it should work properly. Is that update something that would be feasible for you?

One question: are you seeing the sourcemaps listed when accessing http://localhost:5601/api/apm/sourcemaps? (the url corresponds to Kibana, bear in mind that perhaps your Kibana domain is different)

Thanks,
Alberto

[Hamunaptroid]

Hello @Alberto_Delgado,

Thank you,

I can't see anything when I open the link (with my kibana of course). So what does it exactly mean ? Sourcemaps are indexed by Elasticsearch but they are not accessible by kibana ?
I don't understand why I was able to find uploaded sourcemps in Kibana as I showed, but can't see anything on /api/apm/sourcemaps.

We will upgrade ELK stack to version 8.2

[Hamunaptroid]

Hello @Alberto_Delgado,

I am having 2nd thoughts about upgrading our ELK stack to 8.2
I am worried about necessity of using fleet which requires security features enabled.

I am not sure if apm-server is entirely replaced by fleet in 8.2. ...
After apm integration the last step in this documentation is to stop legacy apm server process

Upgrade a self-installation to the APM integration | APM User Guide [8.2] |...

but in step 5. it says at the end to start apm-server (after apmm integration)

Upgrade a self-installation of APM Server standalone to 8.2.0 | APM User...

So apm-server is no longer needed and can be used in 8.2 only to write to "well-defined data streams" (whatever that means). In future releases is apm-server going to be depreciated and removed?

You wrote that if I update elastic to 8.2 it should work. Why it should work in new version? Some time ago, we upgraded elk stack from older versions, I think 6.x to 7.x because of sourcemaps and it did nothing. Were you able to find what exactly is wrong with it now ?

Thank you

[simitt (Silvia Mitter)]

Hi @Hamunaptroid ,
there is no known issue with sourcemaps on 7.17 for standalone APM Server. So it is not required to upgrade to 8.x for using sourcemaps.

As my colleague already pointed out, in 7.17 for standalone APM Server you need to upload the sourcemap via APM Server (not via Kibana). Can you please try again by following these steps to find out where the issue lies:

(0) Start up APM Server with logging level set to debug.
(1) Upload the sourcemap via APM Server and then query it by service.name, service.version and bundle_filepath from Kibana. These three attributes form a unique key. There should only be 1 sourcemap with this key. If you already uploaded multiple of them, please delete the redundant ones.
(2) Ingest an example document matching the service.name, service.version and bundle_filepath (path in the documents stack trace).
For testing purposes, you can reuse a test document , but you need to set the service.name, service.version and path according to the values of your uploaded sourcemap.
You can upload the document for example via curl

curl -i -k -H "Content-type: application/x-ndjson" -H "Authorization: Bearer ${ELASTIC_APM_SECRET_TOKEN}" --data-binary @example.ndjson ${ELASTIC_APM_SERVER_URL}/intake/v2/rum/events

(3) Fetch the specific test document from Elasticsearch and check if sourcemapping has been applied. If so, it shows that there is no general issue, but most certainly a problem with your sourcemap or your data. In this case, please double check that the mentioned keys (service.name, service.version and bundle_filepath==path) are identical in your sourcemap and the ingested error events.

If sourcemapping has not been applied, then please share the apm-server logs, the metadata of your sourcemap and the example document that you uploaded with us and we can take a closer look (remove any sensitive information before sharing).

[Alberto_Delgado]

Hi @Hamunaptroid,

Have you had time to check my colleague comment?

Thanks,
Alberto

[Hamunaptroid]

Hello Alberto,

Yes, I read it. I don't know (yet) how to query it by service.name , service.version and bundle_filepath in Kibana.

Is this part mandatory when uploading sourcemaps to apm-server or It is only needed with security enabled in ELK ?

"Authorization: Bearer ${ELASTIC_APM_SECRET_TOKEN}"

[simitt (Silvia Mitter)]

Yes, I read it. I don't know (yet) how to query it by service.name , service.version and bundle_filepath in Kibana.

You can run a bool quey for querying for multiple fields and their values. The main point with this step (1) is to ensure the sourcemap was processed and indexed to ES with the expected attributes.

Is this part mandatory when uploading sourcemaps to apm-server

You only need to add the authorization part when you have secret_token (or accordingly api_key) enabled in your APM Server.

[Hamunaptroid]

Hello @simitt,

I found out we were still missing "ServiceName" for rum agent config, so during initialization this value was empty, but even after we added the value and reinitialized rum agent with new properties, kibana doesn't show stacktrace. When I query uploaded sourcemap, it has all of the properties it should have - serviceName, ServiceVersion, etc.

In step (3) you suggested uploading "example document" to /intake/v2/rum/events, but in documentation only mentioned endpoint for our version 7.17 is /assets/v1/sourcemaps.

I don't understand what exactly do you mean by uploading document after uploading sourcemap.
This is how I uploaded sourcemap via curl

curl http://mvapmsrv1:8200/assets/v1/sourcemaps -X POST \
  -F 'service_name="fe-lab"' \
  -F 'service_version="v105.7.0"' \
  -F 'bundle_filepath="/tmp/release_unzipped/106/package/build/static/js/js506.chunk.js"' \
  -F 'sourcemap=@506.chunk.js.map'

When I try to upload sourcemaps to /intake/v2/rum/events with same curl, I get following error even if I set Content type.

"message": "invalid content type: 'multipart/form-data; boundary=----------------------------ab9b9b565d06'"

This error changes depending on content type, with content type set to
"application/x-ndjson" I get following:

{
  "accepted": 0,
  "errors": [
    {
      "message": "decode error: data read error: readObjectStart: expect { or n,",
      "document": "------------------------------26199de70c76\r"
    }
  ]
}

So far, the only way to upload sourcemaps to apm-server so elastic can index sourcemaps is upload to /assets/v1/sourcemaps without specifying content type.

[Hamunaptroid]

Ok, so I guess this endpoint is only for the events (it's even in the neme of endpoint but I didn't get it at that time)

/intake/v2/rum/events

I uploaded sourcemap again, but I didn't upload event to this endpoint, I triggered the error "manually" in app (via browser). But Sourcemapping wasn't applied.

I tried to google some test sourcemap that I could upload but I didn't find any.
Do you have any test sourcemap I could upload to test if sourcemap would apply?

I am not sure if this is related, but I see this in Kibana

I can't tell right now if that is an app error or there is something wrong with the sourcemap data.

Thank you

[Alberto_Delgado]

Hi @Hamunaptroid,

Yes, intake/v2/rum/events endpoint is not related to sourcemaps.

I set up an example that I hope helps you:

Versions used on the example:

APM Server: 7.17
RUM agent: 5.12.0

html:

<html>
    <head>
        <title>source map test</title>
    </head>
    <body>
        <button id="error-button">click to generate an error</button>
        <h1>Welcome to the sourcemap test page</h1>
        <script src="https://unpkg.com/@elastic/apm-rum@5.12.0/dist/bundles/elastic-apm-rum.umd.min.js" crossorigin></script>
        <script src="test.min.js"></script>

        <script type="text/javascript">
            document.getElementById('error-button').addEventListener('click', function handleClick() {
                functionInsideTestFile()
            })
            var agent = elasticApm.init({
                serviceName: 'sourcemap_test_7_17',
                serviceVersion: 'sourcemap_test_7_17_version',
                serverUrl:"http://localhost:8200"
            })
    </script>
    </body>
</html>

javascript file before the minify process:

function functionInsideTestFile() {
    throw new Error("I am an error, I have good intentions though.")
}

javascript after the minify process:

function functionInsideTestFile(){throw new Error("I am an error, I have good intentions though.")}
//# sourceMappingURL=http://localhost:1111/sourcemap_test_7_17/test.min.js.map

Command executed to create the minification and map:

npx terser test.js --compress --mangle -o test.min.js --source-map "url='http://localhost:1111/sourcemap_test_7_17/test.min.js.map'"

Note: localhost:1111 corresponds to my local dev server.

Case 1:

If I trigger the error without uploading the sourcemap, we are going to see this on Kibana:

As you can see there, the UI refers to the minified file test.min.js and colNumber 41

--

Now if we want to see the stacktrace related to the original file, what we need to do is to upload the sourcemap (I'm sharing the two common strategies):

Using CURL:

curl -i -X POST http://localhost:8200/assets/v1/sourcemaps \
--header 'Authorization: ApiKey TUVVRWJJRUJoRHdOVXN6WFJRWEw6NF93MGNFQlBTRkdhXzFITy1SZkVpdw==' \
--header 'Content-Type: multipart/form-data' \
--form service_name="sourcemap_test_7_17" \
--form service_version="sourcemap_test_7_17_version" \
--form bundle_filepath="http://localhost:1111/sourcemap_test_7_17/test.min.js" \
--form sourcemap="@test.min.js.map"

USING node.js script

const fs = require('fs')
const request = require('request')


function uploadSourcemap() {
     const headers = {
       'Content-Type': 'multipart/form-data',
       Authorization: "ApiKey TUVVRWJJRUJoRHdOVXN6WFJRWEw6NF93MGNFQlBTRkdhXzFITy1SZkVpdw=="
     }

     const sourcemapPath = 'test.min.js.map'

     const formData = {
       sourcemap: fs.createReadStream(sourcemapPath),
       service_version: 'sourcemap_test_7_17_version',
       bundle_filepath:
         'http://localhost:1111/sourcemap_test_7_17/test.min.js',
       service_name: 'sourcemap_test_7_17'
     }

     request.post(
       {
         url: 'http://localhost:8200/assets/v1/sourcemaps',
         formData,
         headers
       },
       function (err, resp, body) {
         if (err || (resp.statusCode !== 200 && resp.statusCode !== 202)) {
           var message = `Error while uploading sourcemap, error: ${err}, response: ${
             resp && resp.statusCode
           }, body: ${body}`
           console.log(message)
         } else {
           console.log("sourcemap upload properly!")
         }
       }
     )
}

uploadSourcemap();

Okay, now, after doing this. If I go to the webpage and I trigger the error again, that's what I can see on Kibana:

Now we are seeing the UI referring to the unminified file "test.js" and the proper lineno and colno.

By the way, this is how my local project structure looks like:

How to create an api key using Kibana

If you need it, the link below is pretty helpful. I have followed the instructions there when I was working on the example

API Keys | Kibana Guide [master] | Elastic

Thanks,
Alberto

[Hamunaptroid]

Hello @Alberto_Delgado

Thank you for reply, I appreciate the detailed example.
Before I try to simulate these exact instructions, I have question regarding "bundle_filepath".

Is it necessary to point to exact file via url address as you showed me in this example or would it work just by pointing to any directory where this file is situated ?

When I attempted to upload sorcemaps I tried to upload it from different server (not AIX webserver) so bundle_filepath is set just to directory where that file is. When I query sourcemaps in kibana it points to the directory from which I have uploaded it (obviously), but do sourcemaps need to have url address for that specific file ?

"sourcemap" : {
            "bundle_filepath" : "/mnt/t/2/main.min.js",
            "sourcemap" : """{"version":3,"names":["this","webpackJsonpwww","push","e","t","n","d","T","O","S","m","A","p","N","v","j","C","h","y","P","F","M","i","a","o","r","s","c","l","_","I","E","type","b","ON_SIDER_LOADING","payload","isLoading","u","ON_DEFAULT_SUBMENU","subItems","key","ON_ADD_BADGE_COUNT_IS_LOADING","ON_ADD_BADGE_COUNT_DATA","badgeCount","ON_ACTIVE_LINK","item","ON_OPENED_DROPDOWN_ITEM","ON_COLLAPSED_MENU","mqlMatches","ON_MENU_ITEM_CHANGE.....................

If that is the case that would mean I would need to upload it directly from our AIX webserver because otherwise uploading file fails with "http: no such file"

[Alberto_Delgado]

Hello @Hamunaptroid,

bundle_filepath is the final path of of the bundle when it is deployed to production. So, yes, I strongly recommend you to point to exact file via url address like on the example I shared on the thread.

I'll check the snippets we show on our documentation and see if we can do something to highlight that on a better way.

Thanks,
Alberto

[Hamunaptroid]

Hello,

Sourcemaps are working !
Your detailed example for sourcemaps helped me big time with testing and fixing this problem.

Thank you very much @Alberto_Delgado and @simitt for all your help