How can I estimate MbB/s for an API endpoit based on APM logs?

[rturk]

Hi everyone,

I'm working with Elasticsearch APM monitoring performance of our services, and I'm interested in estimating the network traffic (in MB/s) generated by the requests and responses logged by APM for APIs, or Paths.

Any suggestions for how to achieve this?

[trentm (Trent Mick)]

@rturk Thanks for the question.

One way would be to use Packetbeat: Network Analytics Using Elasticsearch | Elastic which I'm pretty sure will provide metrics on total incoming and outgoing bytes over the network. That's independent of the Node.js APM agent.

Using just the Node.js APM agent:

First what the APM agent might provide automatically. The agent does record the "encoded_body_size" on some spans. However, this is only recording the content-length header value from the HTTP response for client HTTP requests, and only for some instrumentations. For example, when the AWS S3 client is used to make a request to S3, the Content-Length of the HTTP response from S3 is recorded.

Because this data is limited to client HTTP requests, an is only recorded by some instrumentations, I think this would be too limited. I'm guessing from your question that you are more interested in the bytes in and out to your service's endpoints.

I think this would require some manual instrumentation. The best current option with the Node.js APM agent would be to use its OpenTelemetry Metrics API support: OpenTelemetry bridge | APM Node.js Agent Reference [4.x] | Elastic

This allows you to the OpenTelemetry Metrics API to define metrics and update them in your code. The APM agent translates those Metrics API calls into metrics that are sent to the APM server and stored in Elasticsearch. You would then need to create your own dashboard/charts in Kibana to visualize those metrics.

If I understand what you want correctly, you might want to define Histogram metrics to capture the HTTP server request body size and response body size. You'd need a handler on your API server object that update each metric on each request (after the request and response body sizes are known).

If it helps at all:

• The OpenTelemetry Semantic Conventions suggest http.server.request.body.size and http.server.response.body.size for the names of these metrics: Semantic Conventions for HTTP Metrics | OpenTelemetry
• You can find an example of using the OpenTelemetry Metrics API to create and update a Histogram metric in the OpenTelemetry http package instrumentation. Search for _httpServerDurationHistogram here: opentelemetry-js/experimental/packages/opentelemetry-instrumentation-http/src/http.ts at main · open-telemetry/opentelemetry-js · GitHub Actually so also here: opentelemetry-js/doc/metrics.md at main · open-telemetry/opentelemetry-js · GitHub

I realize this will be a fair amount of learning and work.