Hi,
I have a OTEL based custom instrumentation for a http server and Elastic Cloud as collector/backend and I want to do head based sampling.
These are my config env config for the same:
ENV OTEL_TRACES_SAMPLER_ARG="0.1"
ENV OTEL_LOG_LEVEL="warn"
ENV OTEL_BSP_MAX_QUEUE_SIZE="75000"
ENV OTEL_BSP_MAX_EXPORT_BATCH_SIZE="75000"
ENV OTEL_BSP_SCHEDULE_DELAY="2500"
ENV OTEL_EXPORTER_OTLP_TRACES_TIMEOUT="30000"
ENV OTEL_EXPORTER_OTLP_TIMEOUT="30000"
ENV OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
ENV OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
This works fine and samples the traces on the application/OTEL agent level, but the calculated metrics in kibana are also sampled. As per Elastic documentation, central configuration for OTEL agents is not supported.
So is there a way I can send sampled traces but still get complete calculated transaction metrics?
Doing tail based sampling is hard because this service is high throughput and when 100% sampled traces are enabled, the agents end up dropping 50% of the traces, and also decreases throughput by 30-50%.
