How to set the "MaxCallRecvMsgSize" of APM server?

[tttoad (ttoad)]

openTelemetry: otlp/elastic
rpc error: code = ResourceExhausted desc = grpc :received message larger than max (5454409 vs. 4194304)
How to set the "MaxCallRecvMsgSize" of APM server?

[axw (Andrew Wilkins)]

The server uses the default size, 4MB, and it can't currently be configured.

Are you using opentelemetry-collector to send data? If so, you might want to set the send_batch_size property to limit the payload size (see opentelemetry-collector/processor/batchprocessor at main · open-telemetry/opentelemetry-collector)

[tttoad (ttoad)]

This configuration does not work.

    send_batch_size: 1
    send_batch_max_size: 2
    timeout: 10s

maybe the metric of "go.opentelemetry.io/contrib/instrumentation/runtime" too large

	// start go pprof metric observation
		if err = runtime.Start(runtimeOption...); err != nil {
			return nil, errors.Wrap(err, "failed to start runtime metric observation")
		}

[axw (Andrew Wilkins)]

maybe the metric of "go.opentelemetry.io/contrib/instrumentation/runtime " too large

Maybe. It seems to me that if the payload is reaching 4MB, then there's probably some runaway payload growth happening either in the OpenTelemetry SDK or OpenTelemetry Collector. I'm reluctant to just go and add configuration to increase the max size without first understanding why this is happening.

Any chance you can provide a small program that reproduces the issue?

[tttoad (ttoad)]

I haven't reproduced the problem yet.
The bug is in the business's application and I don't have access to their code.
When I have time, I'll write a demo to see if I can reproduce it.

[tttoad (ttoad)]

I checked the history logs and found that this error occurred 6 hours after the service was started. There should be a bug causing the data to pile up, not a single metric with too much data.