# Unable to see application logs which are added to a Opentelemetry span in APM

**URL:** <https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794>\
**Category:** APM\
**Tags:** java, open-telemetry\
**Created:** [October 17, 2022, 2:04pm UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794 "2022-10-17T14:04:30Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![User28](https://avatars.discourse-cdn.com/v4/letter/u/7cd45c/32.png) [@User28](https://discuss.elastic.co/u/User28)\
**Post date:** [October 17, 2022, 2:04pm UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/1 "2022-10-17T14:04:30Z")

</div>

**Kibana version** : 7.17.4

**Elasticsearch version** : 7.17.4

**APM Server version** : 7.17.4

**APM Agent language and version** : OpenTelemetry agent v1.18.0

**Browser version** : Chrome

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**:

**Fresh install or upgraded from other version?** fresh

**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. No

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

We are instrumenting spring boot app with Opentelemetry(otel) and with OTLP sending otel data to Elastic APM.  
In APM, we can see traces done on app.

**2 Questions.**

**1.** We are adding app logs to opentelemetry span event, but this event(logs) we can't see to a particular span in APM or Kibana

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/3/d3ce7495519e0ebf4002bdfc8880273f8ce6aa1c.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9351a02fb2538f2045751cef05da448977d8094.png)

**2.**  
App errors are showing in APM Errors but now in Span document in Discover . How to link this?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/3/73960c659dd3f988464265cb872f8b1d0193fea8.png)

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [October 18, 2022, 12:42am UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/2 "2022-10-18T00:42:25Z")

</div>

@User28 how are you running APM Server? Have you made the [switch to the Elastic APM integration](https://www.elastic.co/guide/en/apm/guide/7.17/upgrade-to-apm-integration.html)? This is necessary for span events to show up as logs. Exception span events are a special case that work either way, but other span events will be silently dropped unless you're using the integration, or otherwise if you upgrade to 8.0+.

---

<div class="post-metadata">

**Author:** ![User28](https://avatars.discourse-cdn.com/v4/letter/u/7cd45c/32.png) [@User28](https://discuss.elastic.co/u/User28)\
**Post date:** [October 18, 2022, 11:27am UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/3 "2022-10-18T11:27:35Z")

</div>

Hi @axw APM Server is running on Azure AKS, deployed with helm along with Elasticsearch and Kibana. Please find below apm-server yaml file for the same.

```auto
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: {{ .Release.Name }}
  namespace: efk-dev
spec:
  version: {{ .Values.elasticStackVersion }}
  count: 1
  elasticsearchRef:
    name: {{ .Release.Name }}
  kibanaRef:
    name: {{ .Release.Name }}
  config:
    output:
      elasticsearch:
        username: elastic
        password: elastic
  http:
    tls:
      selfSignedCertificate:
        disabled: true

  podTemplate:
    spec:
      containers:
        - name: apm-server
          resources:
            limits:
              memory: 8Gi
              cpu: 1

```

I have added **APM integration** using this doc [Quick start | APM User Guide [7.17] | Elastic](https://www.elastic.co/guide/en/apm/guide/7.17/apm-quick-start.html#add-apm-integration)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/4/e442aa896e6eeda042455c4957dc92b2cea467ac.png)

But still we **don't see the logs** in Span event metadata or in APM logs. Do we need any more configuration?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/9/898228e1243a86d0cec116dcec347894dddc2f08.png)

Regarding upgrade to 8.0+ version, we may not be able to do in our current dev plans, but yes it's in our roadmap.

---

<div class="post-metadata">

**Author:** ![User28](https://avatars.discourse-cdn.com/v4/letter/u/7cd45c/32.png) [@User28](https://discuss.elastic.co/u/User28)\
**Post date:** [October 18, 2022, 1:57pm UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/4 "2022-10-18T13:57:46Z")

</div>

**Updates:**

After correcting the log indices in ( Observability -\> Logs -\> Settings) logs are appearing in APM Logs

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4dd589f54451ba1740aaf55b0865f20dc7aa3dfa.png)

**But still not in metadata of the Span, I need to see this logs in a single trace with Kibana Discover**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/b/9bb4ecfa1133d8c84783ac8a8d59a09e013d6a3f.png)

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [October 19, 2022, 1:18am UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/5 "2022-10-19T01:18:30Z")

</div>

> [@User28](#):
>
> **But still not in metadata of the Span, I need to see this logs in a single trace with Kibana Discover**

Can you see the log event in Discover if you remove the search for "Workspace GraphQL request received"?

---

<div class="post-metadata">

**Author:** ![User28](https://avatars.discourse-cdn.com/v4/letter/u/7cd45c/32.png) [@User28](https://discuss.elastic.co/u/User28)\
**Post date:** [October 19, 2022, 5:51am UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/6 "2022-10-19T05:51:06Z")

</div>

No ☹  
FYI Java code for adding logs in span event

```auto
final Span currentSpan = Span.current();
        AttributesBuilder builder = Attributes.builder();

        if (currentSpan != null) {

            builder.put("logger", event.getLoggerName())
                    .put("level", event.getLevel().toString())
                    .put("message", event.getFormattedMessage());

            currentSpan.addEvent("LogEvent", builder.build());

            if (Level.ERROR.equals(event.getLevel())) {
                currentSpan.setStatus(StatusCode.ERROR);
            }

            IThrowableProxy throwableProxy = event.getThrowableProxy();
            if (throwableProxy instanceof ThrowableProxy) {
                Throwable throwable = ((ThrowableProxy) throwableProxy).getThrowable();
                if (throwable != null) {
                    currentSpan.recordException(throwable);
                }
            }
        }

```

Discover JSON Span Document for ref:

```auto
{
  "_index": "apm-7.17.4-span-000001",
  "_type": "_doc",
  "_id": "m_y-7oMBmPpmpWN3RIb-",
  "_version": 1,
  "_score": 1,
  "_source": {
    "parent": {
      "id": "88bc9e45ea1b3e7b"
    },
    "agent": {
      "name": "opentelemetry/java",
      "version": "1.18.0"
    },
    "processor": {
      "name": "transaction",
      "event": "span"
    },
    "labels": {
      "process_runtime_description": "Oracle Corporation OpenJDK 64-Bit Server VM 17.0.2+8-86",
      "thread_id": 81,
      "thread_name": "graphql-exec-1",
      "graphql_operation_type": "mutation",
      "graphql_document": "mutation {\n _0_createWorkspace: createWorkspace(newWorkspace: ?) {\n workspace {\n id\n description\n comment\n type\n participants {\n user\n isOwner\n }\n groups {\n entityNumber\n id\n sequenceNumber\n stageId\n entities {\n entityNumber\n id\n iditId\n defPackageId\n domain\n }\n }\n }\n }\n}",
      "telemetry_auto_version": "1.18.0"
    },
    "observer": {
      "hostname": "efk-dev-apm-server-6dc9bcb969-wwg74",
      "id": "f509f864-2f8f-4399-9418-267b5ba06e4a",
      "ephemeral_id": "ff494f46-36cc-4b4f-9fb1-10d6514bde07",
      "type": "apm-server",
      "version": "7.17.4",
      "version_major": 7
    },
    "trace": {
      "id": "07430797357ef24ea7383545285d9819"
    },
    "@timestamp": "2022-10-19T05:36:25.751Z",
    "ecs": {
      "version": "1.12.0"
    },
    "service": {
      "name": "workspace"
    },
    "event": {
      "outcome": "unknown"
    },
    "timestamp": {
      "us": 1666157785751514
    },
    "span": {
      "duration": {
        "us": 980639
      },
      "name": "mutation",
      "id": "4540b33e31271de2",
      "type": "app"
    }
  },
  "fields": {
    "span.name": [
      "mutation"
    ],
    "labels.process_runtime_description": [
      "Oracle Corporation OpenJDK 64-Bit Server VM 17.0.2+8-86"
    ],
    "labels.graphql_operation_type": [
      "mutation"
    ],
    "labels.thread_id": [
      81
    ],
    "labels.graphql_document": [
      "mutation {\n _0_createWorkspace: createWorkspace(newWorkspace: ?) {\n workspace {\n id\n description\n comment\n type\n participants {\n user\n isOwner\n }\n groups {\n entityNumber\n id\n sequenceNumber\n stageId\n entities {\n entityNumber\n id\n iditId\n defPackageId\n domain\n }\n }\n }\n }\n}"
    ],
    "labels.thread_name": [
      "graphql-exec-1"
    ],
    "labels.telemetry_auto_version": [
      "1.18.0"
    ],
    "trace.id": [
      "07430797357ef24ea7383545285d9819"
    ],
    "span.duration.us": [
      980639
    ],
    "processor.event": [
      "span"
    ],
    "agent.name": [
      "opentelemetry/java"
    ],
    "event.outcome": [
      "unknown"
    ],
    "service.name": [
      "workspace"
    ],
    "processor.name": [
      "transaction"
    ],
    "span.id": [
      "4540b33e31271de2"
    ],
    "observer.version_major": [
      7
    ],
    "observer.hostname": [
      "efk-dev-apm-server-6dc9bcb969-wwg74"
    ],
    "span.type": [
      "app"
    ],
    "observer.id": [
      "f509f864-2f8f-4399-9418-267b5ba06e4a"
    ],
    "timestamp.us": [
      1666157785751514
    ],
    "@timestamp": [
      "2022-10-19T05:36:25.751Z"
    ],
    "observer.ephemeral_id": [
      "ff494f46-36cc-4b4f-9fb1-10d6514bde07"
    ],
    "observer.version": [
      "7.17.4"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "parent.id": [
      "88bc9e45ea1b3e7b"
    ],
    "agent.version": [
      "1.18.0"
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [October 19, 2022, 6:49am UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/7 "2022-10-19T06:49:15Z")

</div>

OK. I'm not sure why the exception isn't showing up in Discover, but the `LogEvent` span event isn't showing up due to the version of APM Server.

As I mentioned earlier, in 7.17, log events are silently dropped. You can configure APM Server in 7.17 to write to data streams instead by adding `apm-server.data_streams.enabled: true` to your your apm-server config (i.e. under the `config:` in the k8s YAML you supplied above.) See also [General configuration options | APM User Guide [7.17] | Elastic](https://www.elastic.co/guide/en/apm/guide/7.17/configuration-process.html#_configuration_options_data_streams)

Bear in mind that this will mean that all transaction and span events will start being written to `traces-apm-default`; metrics will be sent to `metrics-apm*-default`, and logs will go to `logs-apm*-default`.

---

<div class="post-metadata">

**Author:** ![User28](https://avatars.discourse-cdn.com/v4/letter/u/7cd45c/32.png) [@User28](https://discuss.elastic.co/u/User28)\
**Post date:** [November 4, 2022, 6:51am UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/8 "2022-11-04T06:51:05Z")

</div>

Thanks @axw . For now we are okay to have logs un Log Tab of APM.  
When we need these logs in discover in future, we will open different ticket.

Thanks for all the support

---

<div class="post-metadata">

**Author:** ![Peter\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/peter_li/32/109125_2.png) [@Peter\_Li](https://discuss.elastic.co/u/Peter_Li)\
**Post date:** [November 11, 2022, 5:52pm UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/9 "2022-11-11T17:52:57Z")

</div>

Hi James...I am also having problem seeing logs in APM transaction/trace sample view. Can you share what Observability -\> Logs -\> Settings changes were needed ?

> After correcting the log indices in ( Observability -\> Logs -\> Settings) logs are appearing in APM Logs

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 2, 2022, 1:53pm UTC](https://discuss.elastic.co/t/unable-to-see-application-logs-which-are-added-to-a-opentelemetry-span-in-apm/316794/10 "2022-12-02T13:53:31Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.