Elastic APM RUM: Can't Upload Sourcemaps

Kibana version: 6.8.1
Elasticsearch version: 6.8.1
APM Server version: 6.8.1
APM Agent language and version: JS RUM "@elastic/apm-rum": "^4.3.1",
Original install method and version:
Yarn 1.7.0
Node 8.14.0
Webpack 4
Symfony Webpack Encore 0.26

Is there anything special in your setup?
I have multiple endpoints in my webpack, so I am trying to upload multiple sourcemaps. For example:
app.js
vendor.js
home.js
search.js

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I have followed these two guides.
https://www.elastic.co/guide/en/apm/agent/js-base/4.x/sourcemap.html
https://www.elastic.co/guide/en/apm/server/6.8/sourcemap-api.html'

When I try to upload a single sourcemap file from cURL I get a 202 Accepted, but the exptected sourcemap index is not created and cannot be found in Kibana (apm-(version)-sourcemaps). The error is also not using the sourcemap.

curl -v -X POST http://apm:8200/assets/v1/sourcemaps   -F service_name="test-js"   -F service_version="314d1f2"   -F bundle_filepath="http://www.local.test/build/amp/search.314d1f2.js"   -F sourcemap=@/var/www/html/symfony/web/build/amp/search.314d1f2.js.map
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 172.18.0.7...
* TCP_NODELAY set
* Connected to apm (172.18.0.7) port 8200 (#0)
> POST /assets/v1/sourcemaps HTTP/1.1
> Host: apm:8200
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Length: 159441
> Content-Type: multipart/form-data; boundary=------------------------d63c0681f0af8df2
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 202 Accepted
< Date: Fri, 09 Aug 2019 20:13:43 GMT
< Content-Length: 0
<
* Connection #0 to host apm left intact

Here is the relevant JSON produced by the APM error caused by a JS error:

{
      "exception": {
        "message": "Uncaught TypeError: Cannot read property 'currentIndex' of undefined",
        "type": "TypeError",
        "stacktrace": [
          {
            "line": {
              "number": 1,
              "column": 8703
            },
            "sourcemap": {
              "updated": false,
              "error": "No Sourcemap available for Service Name: 'test-js', Service Version: '314d1f2' and Path: 'http://www.local.test/build/amp/search.314d1f2.js'."
            },
            "filename": "build/amp/search.314d1f2.js",
            "abs_path": "http://www.local.test/build/amp/search.314d1f2.js",
            "function": "V",
            "library_frame": false,
            "exclude_from_grouping": false
          },
          {
            "abs_path": "http://www.local.test/build/amp/search.314d1f2.js",
            "function": "<anonymous>",
            "library_frame": false,
            "exclude_from_grouping": false,
            "line": {
              "number": 1,
              "column": 6002
            },
            "sourcemap": {
              "updated": false,
              "error": "No Sourcemap available for Service Name: 'test-js', Service Version: '314d1f2' and Path: 'http://www.local.test/build/amp/search.314d1f2.js'."
            },
            "filename": "build/amp/search.314d1f2.js"
          }
        ]
      },
      "culprit": "build/amp/search.314d1f2.js"
    },

According to this docs: https://www.elastic.co/guide/en/apm/server/current/sourcemap-indices.html#sourcemap-example there should be a index:

Source maps are stored in separate indices of the format apm-[version]-sourcemap .

But this is not the case, even though the file is successfully uploaded:

In addition to the cURL command, I have tried following the guide's advice and using Node to upload the sourcemaps.

console.log('Uploading sourcemaps!');
var request = require('request');
var path = require('path');
var fs = require('fs');
var git = require('git-rev-sync')
var serviceVersion = git.short('./../');
var directory = './web/build/amp';
var directoryPath = path.join(__dirname, directory);

//GET MAPPED FILENAMES
var urls = [];
var files = fs.readdirSync(directoryPath);
files.forEach(function(file) {
    if (file.includes(serviceVersion) && file.includes('.map')) {
        urls.push({
            map: file,
            url: file.replace('.map', '')
        });
    }
});

//SUBMIT FILES TO ELASTIC APM
urls.forEach(function(url) {
    var formData = {
        sourcemap: fs.createReadStream(directoryPath + '/' + url.map),
        service_version: serviceVersion,
        bundle_filepath: 'http://www.local.test/build/amp/'+url.url,
        service_name: 'test-js'
    }
    request.post({url: 'http://apm:8200/assets/v1/sourcemaps',formData: formData}, function (err, resp, body) {
        if (err) {
            console.log('Error while uploading sourcemaps!', err)
        } else {
            console.log('Sourcemap uploaded: http://www.local.test/build/amp/'+url.url)
        }
    })
});

The results for this file are as follows, which indicates successful upload.

Sourcemap uploaded: http://www.local.test/build/amp/vendor.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/log.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/local.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/images.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/runtime.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/blog.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/apm.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/app.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/search.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/videos.314d1f2.js
Sourcemap uploaded: http://www.local.test/build/amp/home.314d1f2.js

I am using these two docker images:
image: docker.elastic.co/elasticsearch/elasticsearch:6.8.1
image: docker.elastic.co/apm/apm-server:6.8.1

I don't know where to go from here, or how to further debug why my sourcemaps aren't taking effect even on successful upload.

Thank you!

Hey @Emirii, welcome to the group. Based on the screenshot of the available apm indices, it appears all APM data is directed to apm-%{[beat.version]}-%{+yyyy.MM.dd} index. I expect the following query will return the sourcemaps you have uploaded, can you confirm?

GET apm-*/_search
{
  "query": {
    "term": {
      "processor.event": "sourcemap"
    }
  }
}

If so, dId you intend for that to be the case? The recommended index configuration in 6.8.1 for apm-server.yml is:

output.elasticsearch
    - index: "apm-%{[beat.version]}-sourcemap"                                                                                                                                                                                                
      when.contains:
        processor.event: "sourcemap"

    - index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"

    - index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"

    - index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"

    - index: "apm-%{[beat.version]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"

    - index: "apm-%{[beat.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"

If it is intended, you should configure apm-server to look for sourcemaps in apm-*, replacing the default of apm-*-sourcemap. This can be done in apm-server.yml with apm-server.rum.source_mapping.index_pattern.

1 Like

Thank you for taking the time @gil ! Your query to find the sourcemaps showed me they were indeed all bundled into the one index. I had the configuration all default except for the required (elasticsearch connection info and enable RUM).

I enabled and changed the configuration of server.rum.source_mapping.index_pattern: "apm-*" and RUM was successful at displaying the sourcemapped code! I will also check out your recommended configuration as well.

Excellent, thanks for reporting back. Glad to hear it is sorted out.