Unable to see application logs which are added to a Opentelemetry span in APM

Kibana version: 7.17.4

Elasticsearch version: 7.17.4

APM Server version: 7.17.4

APM Agent language and version: OpenTelemetry agent v1.18.0

Browser version: Chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version? fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. No

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

We are instrumenting spring boot app with Opentelemetry(otel) and with OTLP sending otel data to Elastic APM.
In APM, we can see traces done on app.

2 Questions.

1. We are adding app logs to opentelemetry span event, but this event(logs) we can't see to a particular span in APM or Kibana


2.
App errors are showing in APM Errors but now in Span document in Discover . How to link this?

@User28 how are you running APM Server? Have you made the switch to the Elastic APM integration? This is necessary for span events to show up as logs. Exception span events are a special case that work either way, but other span events will be silently dropped unless you're using the integration, or otherwise if you upgrade to 8.0+.

Hi @axw APM Server is running on Azure AKS, deployed with helm along with Elasticsearch and Kibana. Please find below apm-server yaml file for the same.

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: {{ .Release.Name }}
  namespace: efk-dev
spec:
  version: {{ .Values.elasticStackVersion }}
  count: 1
  elasticsearchRef:
    name: {{ .Release.Name }}
  kibanaRef:
    name: {{ .Release.Name }}
  config:
    output:
      elasticsearch:
        username: elastic
        password: elastic
  http:
    tls:
      selfSignedCertificate:
        disabled: true

  podTemplate:
    spec:
      containers:
        - name: apm-server
          resources:
            limits:
              memory: 8Gi
              cpu: 1

I have added APM integration using this doc Quick start | APM User Guide [7.17] | Elastic

But still we don't see the logs in Span event metadata or in APM logs. Do we need any more configuration?

Regarding upgrade to 8.0+ version, we may not be able to do in our current dev plans, but yes it's in our roadmap.

Updates:

After correcting the log indices in ( Observability -> Logs -> Settings) logs are appearing in APM Logs

But still not in metadata of the Span, I need to see this logs in a single trace with Kibana Discover

Can you see the log event in Discover if you remove the search for "Workspace GraphQL request received"?

No :frowning:
FYI Java code for adding logs in span event

final Span currentSpan = Span.current();
        AttributesBuilder builder = Attributes.builder();

        if (currentSpan != null) {

            builder.put("logger", event.getLoggerName())
                    .put("level", event.getLevel().toString())
                    .put("message", event.getFormattedMessage());

            currentSpan.addEvent("LogEvent", builder.build());

            if (Level.ERROR.equals(event.getLevel())) {
                currentSpan.setStatus(StatusCode.ERROR);
            }

            IThrowableProxy throwableProxy = event.getThrowableProxy();
            if (throwableProxy instanceof ThrowableProxy) {
                Throwable throwable = ((ThrowableProxy) throwableProxy).getThrowable();
                if (throwable != null) {
                    currentSpan.recordException(throwable);
                }
            }
        }

Discover JSON Span Document for ref:

{
  "_index": "apm-7.17.4-span-000001",
  "_type": "_doc",
  "_id": "m_y-7oMBmPpmpWN3RIb-",
  "_version": 1,
  "_score": 1,
  "_source": {
    "parent": {
      "id": "88bc9e45ea1b3e7b"
    },
    "agent": {
      "name": "opentelemetry/java",
      "version": "1.18.0"
    },
    "processor": {
      "name": "transaction",
      "event": "span"
    },
    "labels": {
      "process_runtime_description": "Oracle Corporation OpenJDK 64-Bit Server VM 17.0.2+8-86",
      "thread_id": 81,
      "thread_name": "graphql-exec-1",
      "graphql_operation_type": "mutation",
      "graphql_document": "mutation {\n  _0_createWorkspace: createWorkspace(newWorkspace: ?) {\n    workspace {\n      id\n      description\n      comment\n      type\n      participants {\n        user\n        isOwner\n      }\n      groups {\n        entityNumber\n        id\n        sequenceNumber\n        stageId\n        entities {\n          entityNumber\n          id\n          iditId\n          defPackageId\n          domain\n        }\n      }\n    }\n  }\n}",
      "telemetry_auto_version": "1.18.0"
    },
    "observer": {
      "hostname": "efk-dev-apm-server-6dc9bcb969-wwg74",
      "id": "f509f864-2f8f-4399-9418-267b5ba06e4a",
      "ephemeral_id": "ff494f46-36cc-4b4f-9fb1-10d6514bde07",
      "type": "apm-server",
      "version": "7.17.4",
      "version_major": 7
    },
    "trace": {
      "id": "07430797357ef24ea7383545285d9819"
    },
    "@timestamp": "2022-10-19T05:36:25.751Z",
    "ecs": {
      "version": "1.12.0"
    },
    "service": {
      "name": "workspace"
    },
    "event": {
      "outcome": "unknown"
    },
    "timestamp": {
      "us": 1666157785751514
    },
    "span": {
      "duration": {
        "us": 980639
      },
      "name": "mutation",
      "id": "4540b33e31271de2",
      "type": "app"
    }
  },
  "fields": {
    "span.name": [
      "mutation"
    ],
    "labels.process_runtime_description": [
      "Oracle Corporation OpenJDK 64-Bit Server VM 17.0.2+8-86"
    ],
    "labels.graphql_operation_type": [
      "mutation"
    ],
    "labels.thread_id": [
      81
    ],
    "labels.graphql_document": [
      "mutation {\n  _0_createWorkspace: createWorkspace(newWorkspace: ?) {\n    workspace {\n      id\n      description\n      comment\n      type\n      participants {\n        user\n        isOwner\n      }\n      groups {\n        entityNumber\n        id\n        sequenceNumber\n        stageId\n        entities {\n          entityNumber\n          id\n          iditId\n          defPackageId\n          domain\n        }\n      }\n    }\n  }\n}"
    ],
    "labels.thread_name": [
      "graphql-exec-1"
    ],
    "labels.telemetry_auto_version": [
      "1.18.0"
    ],
    "trace.id": [
      "07430797357ef24ea7383545285d9819"
    ],
    "span.duration.us": [
      980639
    ],
    "processor.event": [
      "span"
    ],
    "agent.name": [
      "opentelemetry/java"
    ],
    "event.outcome": [
      "unknown"
    ],
    "service.name": [
      "workspace"
    ],
    "processor.name": [
      "transaction"
    ],
    "span.id": [
      "4540b33e31271de2"
    ],
    "observer.version_major": [
      7
    ],
    "observer.hostname": [
      "efk-dev-apm-server-6dc9bcb969-wwg74"
    ],
    "span.type": [
      "app"
    ],
    "observer.id": [
      "f509f864-2f8f-4399-9418-267b5ba06e4a"
    ],
    "timestamp.us": [
      1666157785751514
    ],
    "@timestamp": [
      "2022-10-19T05:36:25.751Z"
    ],
    "observer.ephemeral_id": [
      "ff494f46-36cc-4b4f-9fb1-10d6514bde07"
    ],
    "observer.version": [
      "7.17.4"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "parent.id": [
      "88bc9e45ea1b3e7b"
    ],
    "agent.version": [
      "1.18.0"
    ]
  }
}

OK. I'm not sure why the exception isn't showing up in Discover, but the LogEvent span event isn't showing up due to the version of APM Server.

As I mentioned earlier, in 7.17, log events are silently dropped. You can configure APM Server in 7.17 to write to data streams instead by adding apm-server.data_streams.enabled: true to your your apm-server config (i.e. under the config: in the k8s YAML you supplied above.) See also General configuration options | APM User Guide [7.17] | Elastic

Bear in mind that this will mean that all transaction and span events will start being written to traces-apm-default; metrics will be sent to metrics-apm*-default, and logs will go to logs-apm*-default.

Thanks @axw . For now we are okay to have logs un Log Tab of APM.
When we need these logs in discover in future, we will open different ticket.

Thanks for all the support

Hi James...I am also having problem seeing logs in APM transaction/trace sample view. Can you share what Observability -> Logs -> Settings changes were needed ?

After correcting the log indices in ( Observability -> Logs -> Settings) logs are appearing in APM Logs

Thanks

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.