Unable to see application logs which are added to a Opentelemetry span in APM

User28 · October 17, 2022, 2:04pm

Kibana version: 7.17.4

Elasticsearch version: 7.17.4

APM Server version: 7.17.4

APM Agent language and version: OpenTelemetry agent v1.18.0

Browser version: Chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version? fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. No

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

We are instrumenting spring boot app with Opentelemetry(otel) and with OTLP sending otel data to Elastic APM.
In APM, we can see traces done on app.

2 Questions.

1. We are adding app logs to opentelemetry span event, but this event(logs) we can't see to a particular span in APM or Kibana

2.
App errors are showing in APM Errors but now in Span document in Discover . How to link this?

axw · October 18, 2022, 12:42am

@User28 how are you running APM Server? Have you made the switch to the Elastic APM integration? This is necessary for span events to show up as logs. Exception span events are a special case that work either way, but other span events will be silently dropped unless you're using the integration, or otherwise if you upgrade to 8.0+.

User28 · October 18, 2022, 11:27am

Hi @axw APM Server is running on Azure AKS, deployed with helm along with Elasticsearch and Kibana. Please find below apm-server yaml file for the same.

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: {{ .Release.Name }}
  namespace: efk-dev
spec:
  version: {{ .Values.elasticStackVersion }}
  count: 1
  elasticsearchRef:
    name: {{ .Release.Name }}
  kibanaRef:
    name: {{ .Release.Name }}
  config:
    output:
      elasticsearch:
        username: elastic
        password: elastic
  http:
    tls:
      selfSignedCertificate:
        disabled: true

  podTemplate:
    spec:
      containers:
        - name: apm-server
          resources:
            limits:
              memory: 8Gi
              cpu: 1

I have added APM integration using this doc Quick start | APM User Guide [7.17] | Elastic

But still we don't see the logs in Span event metadata or in APM logs. Do we need any more configuration?

Regarding upgrade to 8.0+ version, we may not be able to do in our current dev plans, but yes it's in our roadmap.

User28 · October 18, 2022, 1:57pm

Updates:

After correcting the log indices in ( Observability -> Logs -> Settings) logs are appearing in APM Logs

But still not in metadata of the Span, I need to see this logs in a single trace with Kibana Discover

axw · October 19, 2022, 1:18am

Can you see the log event in Discover if you remove the search for "Workspace GraphQL request received"?

User28 · October 19, 2022, 5:51am

No
FYI Java code for adding logs in span event

final Span currentSpan = Span.current();
        AttributesBuilder builder = Attributes.builder();

        if (currentSpan != null) {

            builder.put("logger", event.getLoggerName())
                    .put("level", event.getLevel().toString())
                    .put("message", event.getFormattedMessage());

            currentSpan.addEvent("LogEvent", builder.build());

            if (Level.ERROR.equals(event.getLevel())) {
                currentSpan.setStatus(StatusCode.ERROR);
            }

            IThrowableProxy throwableProxy = event.getThrowableProxy();
            if (throwableProxy instanceof ThrowableProxy) {
                Throwable throwable = ((ThrowableProxy) throwableProxy).getThrowable();
                if (throwable != null) {
                    currentSpan.recordException(throwable);
                }
            }
        }

Discover JSON Span Document for ref:

{
  "_index": "apm-7.17.4-span-000001",
  "_type": "_doc",
  "_id": "m_y-7oMBmPpmpWN3RIb-",
  "_version": 1,
  "_score": 1,
  "_source": {
    "parent": {
      "id": "88bc9e45ea1b3e7b"
    },
    "agent": {
      "name": "opentelemetry/java",
      "version": "1.18.0"
    },
    "processor": {
      "name": "transaction",
      "event": "span"
    },
    "labels": {
      "process_runtime_description": "Oracle Corporation OpenJDK 64-Bit Server VM 17.0.2+8-86",
      "thread_id": 81,
      "thread_name": "graphql-exec-1",
      "graphql_operation_type": "mutation",
      "graphql_document": "mutation {\n  _0_createWorkspace: createWorkspace(newWorkspace: ?) {\n    workspace {\n      id\n      description\n      comment\n      type\n      participants {\n        user\n        isOwner\n      }\n      groups {\n        entityNumber\n        id\n        sequenceNumber\n        stageId\n        entities {\n          entityNumber\n          id\n          iditId\n          defPackageId\n          domain\n        }\n      }\n    }\n  }\n}",
      "telemetry_auto_version": "1.18.0"
    },
    "observer": {
      "hostname": "efk-dev-apm-server-6dc9bcb969-wwg74",
      "id": "f509f864-2f8f-4399-9418-267b5ba06e4a",
      "ephemeral_id": "ff494f46-36cc-4b4f-9fb1-10d6514bde07",
      "type": "apm-server",
      "version": "7.17.4",
      "version_major": 7
    },
    "trace": {
      "id": "07430797357ef24ea7383545285d9819"
    },
    "@timestamp": "2022-10-19T05:36:25.751Z",
    "ecs": {
      "version": "1.12.0"
    },
    "service": {
      "name": "workspace"
    },
    "event": {
      "outcome": "unknown"
    },
    "timestamp": {
      "us": 1666157785751514
    },
    "span": {
      "duration": {
        "us": 980639
      },
      "name": "mutation",
      "id": "4540b33e31271de2",
      "type": "app"
    }
  },
  "fields": {
    "span.name": [
      "mutation"
    ],
    "labels.process_runtime_description": [
      "Oracle Corporation OpenJDK 64-Bit Server VM 17.0.2+8-86"
    ],
    "labels.graphql_operation_type": [
      "mutation"
    ],
    "labels.thread_id": [
      81
    ],
    "labels.graphql_document": [
      "mutation {\n  _0_createWorkspace: createWorkspace(newWorkspace: ?) {\n    workspace {\n      id\n      description\n      comment\n      type\n      participants {\n        user\n        isOwner\n      }\n      groups {\n        entityNumber\n        id\n        sequenceNumber\n        stageId\n        entities {\n          entityNumber\n          id\n          iditId\n          defPackageId\n          domain\n        }\n      }\n    }\n  }\n}"
    ],
    "labels.thread_name": [
      "graphql-exec-1"
    ],
    "labels.telemetry_auto_version": [
      "1.18.0"
    ],
    "trace.id": [
      "07430797357ef24ea7383545285d9819"
    ],
    "span.duration.us": [
      980639
    ],
    "processor.event": [
      "span"
    ],
    "agent.name": [
      "opentelemetry/java"
    ],
    "event.outcome": [
      "unknown"
    ],
    "service.name": [
      "workspace"
    ],
    "processor.name": [
      "transaction"
    ],
    "span.id": [
      "4540b33e31271de2"
    ],
    "observer.version_major": [
      7
    ],
    "observer.hostname": [
      "efk-dev-apm-server-6dc9bcb969-wwg74"
    ],
    "span.type": [
      "app"
    ],
    "observer.id": [
      "f509f864-2f8f-4399-9418-267b5ba06e4a"
    ],
    "timestamp.us": [
      1666157785751514
    ],
    "@timestamp": [
      "2022-10-19T05:36:25.751Z"
    ],
    "observer.ephemeral_id": [
      "ff494f46-36cc-4b4f-9fb1-10d6514bde07"
    ],
    "observer.version": [
      "7.17.4"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "parent.id": [
      "88bc9e45ea1b3e7b"
    ],
    "agent.version": [
      "1.18.0"
    ]
  }
}

axw · October 19, 2022, 6:49am

OK. I'm not sure why the exception isn't showing up in Discover, but the LogEvent span event isn't showing up due to the version of APM Server.

As I mentioned earlier, in 7.17, log events are silently dropped. You can configure APM Server in 7.17 to write to data streams instead by adding apm-server.data_streams.enabled: true to your your apm-server config (i.e. under the config: in the k8s YAML you supplied above.) See also General configuration options | APM User Guide [7.17] | Elastic

Bear in mind that this will mean that all transaction and span events will start being written to traces-apm-default; metrics will be sent to metrics-apm*-default, and logs will go to logs-apm*-default.

User28 · November 4, 2022, 6:51am

Thanks @axw . For now we are okay to have logs un Log Tab of APM.
When we need these logs in discover in future, we will open different ticket.

Thanks for all the support

Peter_Li · November 11, 2022, 5:52pm

Hi James...I am also having problem seeing logs in APM transaction/trace sample view. Can you share what Observability -> Logs -> Settings changes were needed ?

After correcting the log indices in ( Observability -> Logs -> Settings) logs are appearing in APM Logs

Thanks

system · December 2, 2022, 1:53pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
OpenTelemetry span events are not visible in APM UI APM nodejs , server , open-telemetry	8	1643	October 10, 2022
APM server : Log events filtered out but not spans APM server	3	281	November 14, 2023
Opentelemetry agent unable to connect APM Server over secure connection APM server	9	6864	August 24, 2021
Failed to push trace data via OTLP exporter: rpc error: code = Unavailable desc = connection closed APM open-telemetry	2	3577	October 28, 2021
Elastic APM does not support the Otel Span Events. is there an alternative? APM java , server	2	749	April 22, 2021

Unable to see application logs which are added to a Opentelemetry span in APM

Related topics