Elastic APM RUM: Can't Upload Sourcemaps

Kibana version: 6.8.1
Elasticsearch version: 6.8.1
APM Server version: 6.8.1
APM Agent language and version: JS RUM "@elastic/apm-rum": "^4.3.1",
Original install method and version:
Yarn 1.7.0
Node 8.14.0
Webpack 4
Symfony Webpack Encore 0.26

Is there anything special in your setup?
I have multiple endpoints in my webpack, so I am trying to upload multiple sourcemaps. For example:
app.js
vendor.js
home.js
search.js

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I have followed these two guides.
https://www.elastic.co/guide/en/apm/agent/js-base/4.x/sourcemap.html
https://www.elastic.co/guide/en/apm/server/6.8/sourcemap-api.html'

When I try to upload a single sourcemap file from cURL I get a 202 Accepted, but the exptected sourcemap index is not created and cannot be found in Kibana (apm-(version)-sourcemaps). The error is also not using the sourcemap.

curl -v -X POST http://apm:8200/assets/v1/sourcemaps   -F service_name="test-js"   -F service_version="314d1f2"   -F bundle_filepath="http://www.local.test/build/amp/search.314d1f2.js"   -F sourcemap=@/var/www/html/symfony/web/build/amp/search.314d1f2.js.map
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 172.18.0.7...
* TCP_NODELAY set
* Connected to apm (172.18.0.7) port 8200 (#0)
> POST /assets/v1/sourcemaps HTTP/1.1
> Host: apm:8200
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Length: 159441
> Content-Type: multipart/form-data; boundary=------------------------d63c0681f0af8df2
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 202 Accepted
< Date: Fri, 09 Aug 2019 20:13:43 GMT
< Content-Length: 0
<
* Connection #0 to host apm left intact

Here is the relevant JSON produced by the APM error caused by a JS error:

{
      "exception": {
        "message": "Uncaught TypeError: Cannot read property 'currentIndex' of undefined",
        "type": "TypeError",
        "stacktrace": [
          {
            "line": {
              "number": 1,
              "column": 8703
            },
            "sourcemap": {
              "updated": false,
              "error": "No Sourcemap available for Service Name: 'test-js', Service Version: '314d1f2' and Path: 'http://www.local.test/build/amp/search.314d1f2.js'."
            },
            "filename": "build/amp/search.314d1f2.js",
            "abs_path": "http://www.local.test/build/amp/search.314d1f2.js",
            "function": "V",
            "library_frame": false,
            "exclude_from_grouping": false
          },
          {
            "abs_path": "http://www.local.test/build/amp/search.314d1f2.js",
            "function": "<anonymous>",
            "library_frame": false,
            "exclude_from_grouping": false,
            "line": {
              "number": 1,
              "column": 6002
            },
            "sourcemap": {
              "updated": false,
              "error": "No Sourcemap available for Service Name: 'test-js', Service Version: '314d1f2' and Path: 'http://www.local.test/build/amp/search.314d1f2.js'."
            },
            "filename": "build/amp/search.314d1f2.js"
          }
        ]
      },
      "culprit": "build/amp/search.314d1f2.js"
    },

According to this docs: https://www.elastic.co/guide/en/apm/server/current/sourcemap-indices.html#sourcemap-example there should be a index:

Source maps are stored in separate indices of the format apm-[version]-sourcemap .

But this is not the case, even though the file is successfully uploaded:

In addition to the cURL command, I have tried following the guide's advice and using Node to upload the sourcemaps.

console.log('Uploading sourcemaps!');
var request = require('request');
var path = require('path');
var fs = require('fs');
var git = require('git-rev-sync')
var serviceVersion = git.short('./../');
var directory = './web/build/amp';
var directoryPath = path.join(__dirname, directory);

//GET MAPPED FILENAMES
var urls = [];
var files = fs.readdirSync(directoryPath);
files.forEach(function(file) {
    if (file.includes(serviceVersion) && file.includes('.map')) {
        urls.push({
            map: file,
            url: file.replace('.map', '')
        });
    }
});

//SUBMIT FILES TO ELASTIC APM
urls.forEach(function(url) {
    var formData = {
        sourcemap: fs.createReadStream(directoryPath + '/' + url.map),
        service_version: serviceVersion,
        bundle_filepath: 'http://www.local.test/build/amp/'+url.url,
        service_name: 'test-js'
    }
    request.post({url: 'http://apm:8200/assets/v1/sourcemaps',formData: formData}, function (err, resp, body) {
        if (err) {
            console.log('Error while uploading sourcemaps!', err)
        } else {
            console.log('Sourcemap uploaded: http://www.local.test/build/amp/'+url.url)
        }
    })
});

The results for this file are as follows, which indicates successful upload.

Sourcemap uploaded: http://www.local.test/build/amp/vendor.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/log.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/local.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/images.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/runtime.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/blog.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/apm.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/app.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/search.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/videos.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/home.314d1f2.js

I am using these two docker images:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.1
image: docker.elastic.co/apm/apm-server:6.8.1

I don't know where to go from here, or how to further debug why my sourcemaps aren't taking effect even on successful upload.

Thank you!

Hey @Emirii, welcome to the group. Based on the screenshot of the available apm indices, it appears all APM data is directed to apm-%{[beat.version]}-%{+yyyy.MM.dd} index. I expect the following query will return the sourcemaps you have uploaded, can you confirm?

GET apm-*/_search
{
  "query": {
    "term": {
      "processor.event": "sourcemap"
    }
  }
}

If so, dId you intend for that to be the case? The recommended index configuration in 6.8.1 for apm-server.yml is:

output.elasticsearch
    - index: "apm-%{[beat.version]}-sourcemap"                                                                                                                                                                                                
      when.contains:
        processor.event: "sourcemap"

    - index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"

    - index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"

    - index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"

    - index: "apm-%{[beat.version]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"

    - index: "apm-%{[beat.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"

If it is intended, you should configure apm-server to look for sourcemaps in apm-*, replacing the default of apm-*-sourcemap. This can be done in apm-server.yml with apm-server.rum.source_mapping.index_pattern.

1 Like

Thank you for taking the time @gil ! Your query to find the sourcemaps showed me they were indeed all bundled into the one index. I had the configuration all default except for the required (elasticsearch connection info and enable RUM).

I enabled and changed the configuration of server.rum.source_mapping.index_pattern: "apm-*" and RUM was successful at displaying the sourcemapped code! I will also check out your recommended configuration as well.

Excellent, thanks for reporting back. Glad to hear it is sorted out.

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.