APM Integration for Elastic Agent gives 404 accessing "/opentelemetry.proto.collector.metrics.v1.metricsservice/export"

Elastic Observability APM
1

We've just upgraded our APM server from 7.15.1 via standalone install, to an APM Server integration via Elastic Agent.

Our system is on 8.0.1, as is the Elastic Agent.

We had success with this on our development system, that's using the APM integration just fine.
When we do the same in our production environment, APM data isn't appearing in Kibana at all, but instead our logs are just spammed continuously with this:

{"log.level":"error","@timestamp":"2022-03-23T14:44:30.323-0400","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"404 page not found","service.name":"apm-server","url.original":"/opentelemetry.proto.collector.metrics.v1.MetricsService/Export","http.request.method":"POST","user_agent.original":"grpc-dotnet/2.32.0.0","source.address":"10.1.30.118","http.request.id":"55067305-9a16-4be1-84f8-a8fd82def6cf","event.duration":0,"http.response.status_code":404,"error.message":"404 page not found","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2022-03-23T14:44:30.344-0400","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"404 page not found","service.name":"apm-server","url.original":"/opentelemetry.proto.collector.metrics.v1.MetricsService/Export","http.request.method":"POST","user_agent.original":"grpc-dotnet/2.32.0.0","source.address":"10.1.30.131","http.request.id":"e2e6b973-fc10-4f70-8395-7a140456fdad","event.duration":0,"http.response.status_code":404,"error.message":"404 page not found","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2022-03-23T14:44:30.348-0400","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"404 page not found","service.name":"apm-server","url.original":"/opentelemetry.proto.collector.metrics.v1.MetricsService/Export","http.request.method":"POST","user_agent.original":"grpc-dotnet/2.32.0.0","source.address":"10.1.30.118","http.request.id":"780700fe-1359-4ab8-aa1d-5eaaa6907742","event.duration":0,"http.response.status_code":404,"error.message":"404 page not found","ecs.version":"1.6.0"}

I'm guessing the integration needs access to some server, whether the fleet or kibana server I'm not sure. I may have to open up a port in a firewall but I'm not sure what.

2

Did you change anything else, e.g. were you using TLS before and now you're not?

This sounds very much like APM returns a 404 when attempting to send OTLP data · Issue #6267 · elastic/apm-server · GitHub - please take a look and see if any of that applies.

3

Thank you for pointing us in the right direction.

We did wind up updating the versions of TLS that the APM Server integration used, and are now getting APM data.

The errors pointed out above are still happening, but it doesn't stop the bulk of our telemetry - only the handful of applications using OpenTel. We're working on fixing that up with how that team uses their OpenTel agent.

Thanks again.

1 Like
4

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.