Source Maps are causing slice out of bounds exception

[Gegenki (Jordan Douglas)]

My APM server had stopped running shortly after uploading source maps for my node application.

I tried to run my node server again and the APM server crashed again.
I used Kibana to delete the source map index.
When my application client calls apm.captureError my Elastic APM logs show

|2019-01-10T14:47:05.852Z|WARN|[stacktrace]|model/stacktrace.go:88|No Sourcemap available for Service Name: 'Client Dev', Service Version: '1.80.1d' and Path: 'http://127.0.0.1:3000/6fa5db8ad3da1080fe32f149cf5e2ba2bca689dc.js?meteor_js_resource=true'.|
|---|---|---|---|---|
|2019-01-10T14:47:05.853Z|WARN|[stacktrace]|model/stacktrace.go:88|No Sourcemap available for Service Name: 'Client Dev', Service Version: '1.80.1d' and Path: 'http://127.0.0.1:3000/6fa5db8ad3da1080fe32f149cf5e2ba2bca689dc.js?meteor_js_resource=true'.|

Which is largely ok - But when I upload the source maps again I get the following error instead:

panic: runtime error: slice bounds out of range

goroutine 29 [running]:

github.com/elastic/apm-server/sourcemap.subSlice(...)

/go/src/github.com/elastic/apm-server/sourcemap/mapper.go:112

github.com/elastic/apm-server/sourcemap.(*SmapMapper).Apply(0xc4202f3a20, 0xc4207aa100, 0x14, 0xc4206d80f0, 0x7, 0xc42026ee40, 0x59, 0x9, 0x2f388, 0x16fd380, ...)

/go/src/github.com/elastic/apm-server/sourcemap/mapper.go:87 +0x743

github.com/elastic/apm-server/model.(*StacktraceFrame).applySourcemap(0xc420411340, 0x18505a0, 0xc4202f3a20, 0xc420464230, 0x176222d, 0xb, 0x176222d, 0xb, 0xc42059c420, 0xae)

/go/src/github.com/elastic/apm-server/model/stacktrace_frame.go:166 +0x128

github.com/elastic/apm-server/model.(*Stacktrace).Transform(0xc420434930, 0xc42018a180, 0x7, 0x1519e80, 0x0)

/go/src/github.com/elastic/apm-server/model/stacktrace.go:80 +0x1de

github.com/elastic/apm-server/model/error.(*Event).addException(0xc42037f880, 0xc42018a180)

/go/src/github.com/elastic/apm-server/model/error/event.go:283 +0x22c

github.com/elastic/apm-server/model/error.(*Event).fields(0xc42037f880, 0xc42018a180, 0xc4202c45b0)

/go/src/github.com/elastic/apm-server/model/error/event.go:220 +0x91

github.com/elastic/apm-server/model/error.(*Event).Transform(0xc42037f880, 0xc42018a180, 0x9, 0x17602d8, 0x9)

/go/src/github.com/elastic/apm-server/model/error/event.go:193 +0xc9

github.com/elastic/apm-server/publish.(*publisher).processPendingReq(0xc420086180, 0xc4202ee320, 0x2, 0x2, 0xc42018a180, 0x0)

/go/src/github.com/elastic/apm-server/publish/pub.go:147 +0xe7

github.com/elastic/apm-server/publish.(*publisher).run(0xc420086180)

/go/src/github.com/elastic/apm-server/publish/pub.go:134 +0xcc

created by github.com/elastic/apm-server/publish.NewPublisher

/go/src/github.com/elastic/apm-server/publish/pub.go:91 +0x1be

Not sure what I can do from here

[jalvz (Juan Alvarez)]

Ok, that doesn't look so good. I will investigate and follow up with this - most likely it will require a fix on our side.

In the meanwhile, did this happen only after you removed the sourcemap index? And may I ask why you removed the index?

[Gegenki (Jordan Douglas)]

The problem happened before I removed the index.

Yesterday I was having problems uploading the source maps into the system. Then - even after uploading source maps I couldn't get the APM server to find the source map when logging errors.

The APM server crashed randomly today (it has been running for months with no problems). My assumption was that it had finally found the source map but there was a problem processing it.

I removed the index to confirm.

When I removed the index it did not crash - and instead said it couldn't find the source map.
When I uploaded the source map again - it crashed.

1 Things co consider:

1. There could be an issue with the size of my source map.
   When I attempted to upload the source map via my public address it looks like my nginx server rejected it as the size was to big. The source map is generated on the same system as my APM server so I just send it internally (instead of via the internet). The map is 2.4MB

[jalvz (Juan Alvarez)]

Thanks for the info, that is helpful.
One typical reason for sourcemap not found is that the production code was updated and the corresponding, uploaded sourcemap was not re-uploaded, so the code and the sourcemap are out of sync. If there are several sourcemaps with the same service name and version, the most recently uploaded is used.

From the stacktrace it looks indeed that the sourcemap was found and we have an off-by-one bug. As said, I will follow up shortly.

Sorry for the trouble, Apm Server crashing is too bad.

[jalvz (Juan Alvarez)]

Hello, a fix for that has been pushed and will be in in the upcoming 6.6 release.

I still suspect that the sourcemap and the code are out of sync (ie, sourcemap is mapping to a different code version than the one running).

Sourcemap issues are often hard to troubleshoot, I'm sorry that you found this problem.

[Gegenki (Jordan Douglas)]

Excellent. Do you guys have a calendar?
I tried to build it myself but know nothing about Go

[jalvz (Juan Alvarez)]

Hi, unfortunately I do not have a date, but it will be very soon.

If you want to build it yourself, this are the steps, roughly:

1 - Install latest Go version, Gcc and Python virtualenv
2 - Clone and check out the 6.6 branch of github.com/elastic/apm-server
3 - Review https://github.com/elastic/apm-server/blob/6.6/changelogs/head.asciidoc
4 - Install and run Mage:
go get -v -u github.com/magefile/mage
(cd $GOPATH/src/github.com/magefile/mage && go run bootstrap.go)
5 - Update your PATH env var to PATH=$PATH:$GOPATH/bin
6 - From the apm-server directory run make && make update