OTEL Traces not appearing in Observability-Services

Elastic Stack Kibana
1

Posted in Elastic Slack too..

I'm struggling to get OTEL traces from a python application, that are being send to a hosted elastic, to be displayed in observability->services.The application is running in AWS ECS and sends signals via OTLP to a EDOT collector running as a daemonset.

I can see the traces in the traces-generic.otel-default datastream so can confirm end-to-end reception. However the service don't appear in the list of services.

I have trace appearing from other services (NodeJS) than follow a similar route into Elastic that are appearing. These are k8s based, however the plumbing in the collectors is pretty much the same.

2

EDOT Collector Config;

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Collect own metrics
  prometheus:
    config:
      scrape_configs:
        - job_name: "otel-collector"
          scrape_interval: 60s
          static_configs:
            - targets: ["0.0.0.0:8888"]

processors:
  batch:
  batch/metrics:
    send_batch_max_size: 0
    timeout: 1s
  resourcedetection/system:
    detectors: ["system", "ec2"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: false
        host.arch:
          enabled: true
        host.ip:
          enabled: true
        host.mac:
          enabled: true
        host.cpu.vendor.id:
          enabled: true
        host.cpu.family:
          enabled: true
        host.cpu.model.id:
          enabled: true
        host.cpu.model.name:
          enabled: true
        host.cpu.stepping:
          enabled: true
        host.cpu.cache.l2.size:
          enabled: true
        os.description:
          enabled: true
        os.type:
          enabled: true
    ec2:
      resource_attributes:
        host.name:
          enabled: false
        host.id:
          enabled: true
  resource/cloud:
    attributes:
      - key: cloud.instance.id
        from_attribute: host.id
        action: insert
  cumulativetodelta:
    initial_value: auto
    include:
      metrics:
        - db.client.operation.duration
        - http.client.request.duration
        - http.server.request.duration
        - otelcol.processor.batch.batch.send.size
        - otelcol.elasticsearch.bulk_requests.latency
        - otelcol_processor_batch_batch_send_size
        - otelcol_exporter_queue_batch_send_size
      match_type: strict
  elastictrace: {}

exporters:
  debug:
    verbosity: basic
  elasticsearch/otel:
    endpoints:
      - ${env:ELASTICSEARCH_ENDPOINT}
    api_key: ${env:ELASTICSEARCH_API_KEY}
    mapping:
      mode: otel

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, elastictrace]
      exporters: [elasticsearch/otel, debug]

    metrics:
      receivers: [otlp, prometheus]
      processors:
        - batch/metrics
        - resourcedetection/system
        - resource/cloud
        - cumulativetodelta
      exporters: [elasticsearch/otel, debug]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [elasticsearch/otel, debug]

  extensions: [health_check, pprof]