Elastic APM does not support the Otel Span Events. is there an alternative?

[ayada_yassine]

I want to use span events to attach my logs to my span,
Elastic Apm ignore opentelemetry exported span events.

is there an alternative ?

OpenTelemetry Span Events: opentelemetry-specification/api.md at main · open-telemetry/opentelemetry-specification · GitHub

[cyrilleleclerc]

Hello Ayada,

It's correct that we don't support span events at the moment.
Regarding logs integration, we recommend an alternate pattern injecting the trace_id and span_id in the logs and then ingest the logs as a separate stream in Elastic Observability.

1. Use the capability in Otel APM Agents to inject span_id and trace_id in the logging framework such as Log4j2 or Logback/SLF4J in Java.
2. Export you logs as JSON. In Java with Logback, you can use the library logback-json-classic
3. Use a log collector like Elastic Filebeat to collect the logs and ship them in Elastic complying with the Elastic Common Schema to get the trace_id and schema_id recognized
4. When looking at traces in Elastic Observability, use the logs tab to view the logs associated with your trace.

Example log message emitted by Logback with logback-json-classic

{
   "timestamp":"1615986569752",
   "level":"INFO",
   "thread":"http-nio-8081-exec-7",
   "mdc":{
      "trace_id":"98600b7e3ed8be404fed0671cde6448d",
      "trace_flags":"01",
      "span_id":"5f928785b1a2ef30"
   },
   "logger":"com.mycompany.antifraud.AntiFraudController",
   "message":"checkOrder(totalPrice: 005, shippingCountry: US, customerIpAddress: 127.0.0.1): fraudScore: -52, rejected: false, expectedSleep: 53ms, actualSleep: 181ms, delta:241%",
   "context":"default"
}

Logback configuration

cyrille-leclerc/my-shopping-cart/blob/e3f62c2116f238915de6be9635ecb54937d5abec/anti-fraud-java/src/main/resources/logback-spring-json-file-appender.xml

Maven pom.xml

<dependency>
    <groupId>ch.qos.logback.contrib</groupId>
    <artifactId>logback-json-classic</artifactId>
</dependency>
<dependency>
    <groupId>ch.qos.logback.contrib</groupId>
    <artifactId>logback-jackson</artifactId>
</dependency>

Elastic Filebeat Configuration

filebeat.inputs:
  - type: log
    enabled: true
    json.keys_under_root: true
    json.overwrite_keys: true
    paths:
      - /usr/local/var/log/my-shopping-cart/anti-fraud.log
    fields:
      service.name: frontend
    fields_under_root: true

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - timestamp:
      field: "timestamp"
      layouts:
        - "UNIX_MS"
      ignore_missing: true
      test:
        - "1615986552269"
  - rename:
      fields:
        - from: "level"
          to: "log.level"
        - from: "thread"
          to: "process.thread.name"
        - from: "mdc.trace_id"
          to: "trace.id"
        - from: "mdc.span_id"
          to: "span.id"
        - from: "logger"
          to: "log.logger"
      ignore_missing: true
      fail_on_error: true
...