APM-Server unable to fetch sourcemaps from Kibana

Elastic Observability APM
,
1

I'm trying to instruct our standalone ("legacy") APM-Server to get the JavaScript sourcemaps from Kibana, which I've already uploaded. Here is the relevant APM-Server config:

---
apm-server:
  ...

  kibana:
    enabled: true
    host: "https://PRIVATE-NETWORK-IP:443"
    api_key: "FOO:BAR"

    ssl.enabled: "True"
    ssl.verification_mode: "full"
    ssl.certificate_authorities: ["/usr/share/ca-certificates/elastic/ca.crt"]

We're using self-signed certificates for TLS-communication.

With this config, the stacktrace of new JS-errors does not change, so the apm-server does not get the sourcemaps.
In the APM error documents I see the following message in the field error.exception.stacktrace.sourcemap.error :

"Get \"[https://PRIVATE-NETWORK-IP:443/api/apm/sourcemaps](https://private-network-ip/api/apm/sourcemaps)\": x509: certificate signed by unknown authority"

I don't get the error message since I added the certificate authority in the config.
Furthermore: the ssl-section is the exactly same as the one for the elasticsearch output which works fine...

Does anyone know how to fix this?

Stack-Version: 8.6.1

2

Is the CA that signed the Kibana and Elasticsearch certificates exactly the same?

Also, not answering the question, but FYI from 8.7.0 onwards APM Server does not need access to Kibana to pull down source maps; it can fetch them from Elasticsearch instead.

3

@axw thanks for your quick reply, Andrew.

Yes, the CA is the same for all certificates (Kibana & Elasticsearch).

I got the certificate's details with

openssl x509 -in ./CERT-NAME.crt -text -noout

and compared the X509v3 Authority Key Identifier-value of all Elasticsearch & Kibana certificates with the X509v3 Subject Key Identifier-value of the CA certificate and they all match.

Regards.

4

Regarding that with 8.7.0 APM-Server can fetch sourcemaps from Elasticsearch:

This is quite confusing to me.
Here it states that new sourcemaps have to be fetched from Kibana (implying that previously the sourcemaps were fetched from Elasticsearch) and now it goes back to Elasticsearch again?
Furthermore: is this also changing the way to upload the sourcemaps? Is there an Elasticsearch API endpoint for it? Or do I still have to upload it to Kibana and some magic in the background creates an index/data streams? The documenation could be improved as far as I'm concerned.

Regards.

5

I'm not sure what's going on with the TLS certificates I'm afraid. I'll run a test on Monday to confirm that APM Server 8.6.1 does honour the apm-server.kibana.ssl.certificate_authorities config.

Apologies, the comment in apm-server.yml is outdated. I'll open an issue to fix this.

Furthermore: is this also changing the way to upload the sourcemaps? Is there an Elasticsearch API endpoint for it? Or do I still have to upload it to Kibana and some magic in the background creates an index/data streams?

No change to the upload process. Uploading the sourcemap to Kibana will lead to it being indexed as a document in Elasticsearch. APM Server polls Elasticsearch for the current set of sourcemaps stored in Elasticsearch, and fetches them from there.

The documenation could be improved as far as I'm concerned.

Yes, I think you're right. I think it would be good to have a guided walkthrough for this. i.e. showing how to upload a sourcemap for an instrumented application, and verify that it gets applied. I'll open another issue for that, but that's probably not such an easy lift.

6

Yes, I think you're right. I think it would be good to have a guided walkthrough for this. i.e. showing how to upload a sourcemap for an instrumented application, and verify that it gets applied. I'll open another issue for that, but that's probably not such an easy lift.

I forgot that we already have that :sweat_smile:

@apt-get_install_skil which things would you like to see added to that page?

7

Thanks, I'd really appreciate if you could check if the settings get honored and get back to this topic next week.

Regarding the documentation:
The documentation of the upload process is good enough. In my opinion, it would be great to mention what happens in the background after the upload (index gets created which is the source for fetching the sourcemaps from the apm-server's perspective). And of course updating the outdated comments in the reference config file.

Thank you, Andrew.

Regards.

1 Like
8

Thanks, I'd really appreciate if you could check if the settings get honored and get back to this topic next week.

I have confirmed that apm-server.kibana.ssl configuration is honoured. I used ECK (Elastic Cloud for Kubernetes) to do this: it creates separate self-signed CAs for Elasticsearch and Kibana, and configures APM Server with those. I then verified that APM Server could communicate with Kibana; and modified the configuration to remove the CA cert, and verified that it started failing.

So I'm afraid I don't know why things aren't working for you. I guess as a next step, I'd try shelling into the pod and using curl to verify the certs.

Regarding the documentation:
The documentation of the upload process is good enough. In my opinion, it would be great to mention what happens in the background after the upload (index gets created which is the source for fetching the sourcemaps from the apm-server's perspective). And of course updating the outdated comments in the reference config file.

Sounds reasonable. I think we would probably have a disclaimer that these are implementation details that may change.

9

FYI, I have opened these issues to improve docs:

2 Likes
10

Thank's for your efforts, Andrew.

I guess I'll upgrade our stack to the latest version and see if fetching the sourcemaps from Elasticsearch indices solves the issue.

I'm planning to do the upgrade within the next weeks so I can post the results in this topic until it gets closed.

Have a good one.

Best regards,
Lukas

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.