Hello elastic team, I want to get some help on dapr + apmserver

Elastic Observability APM
1

Our service is based on dapr framework, dapr includes tracing function (based on opentelemetry implementation). I want to replace the dapr sidecar with the elastic apm agent, and then still use the apm server for storage.

According to elastic documentation, there should be two ways to do it

  1. Use OpenTelemetry collector to forward to elastic otlp endpoint
  2. The service sends the data directly to the apm server

For the first method, I read the dapr documentation, dapr provides the function of exporting to the opentelemetry collector, but the collector carries zipkin, then can I replace zipkin with apm server? How to do it? （Only zipkin forwarding can be configured in dapr.config）

image
image1143×544 53.8 KB

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
  namespace: default
spec:
  tracing:
    samplingRate: "1"
    zipkin:                 <<<<<<<<<<< Only zipkin can be written here
      endpointAddress: "http://otel-collector.default.svc.cluster.local:9411/api/v2/spans"

For the second method, I did not expect the answer. Is there any relevant practice to learn?

dapr documentation

https://docs.dapr.io/operations/monitoring/tracing/open-telemetry-collector/

elastic documentation

OpenTelemetry integration | APM User Guide [8.11] | Elastic

Finally, I still have some doubts, I got an error when I started opentelemetry collector
'pipeline traces references receiver "otlp/elastic" which does not exist'

"otlp/elastic" is from https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticexporter. If I want to use it, how should I install it?

Anyway, everything is a bit messy, I don’t understand.

2

My dapr.config used

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
  namespace: default
spec:
  tracing:
    samplingRate: "1"
    zipkin:
      endpointAddress: "http://otel-collector:4317"

Then the collector forwards the data to the apm server


apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  labels:
    app: opentelemetry
    component: otel-collector-conf
data:
  otel-collector-config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    extensions:
      health_check:
    exporters:
      logging:
        loglevel: warn
      otlp/elastic: 
        endpoint: "apmserver:8200"
	service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [logging, otlp/elastic]
      metrics:
          receivers: [otlp]
          exporters: [logging, otlp/elastic]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  labels:
    app: opentelemetry
    component: otel-collector
spec:
  replicas: 1  # scale out based on your usage
  selector:
    matchLabels:
      app: opentelemetry
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib-dev:latest
        command:
          - "/otelcontribcol"
          - "--config=/conf/otel-collector-config.yaml"
        resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 200m
            memory: 400Mi
        ports:
          - containerPort: 9411 # Default endpoint for Zipkin receiver.
        volumeMounts:
          - name: otel-collector-config-vol
            mountPath: /conf
        livenessProbe:
          httpGet:
            path: /
            port: 13133
        readinessProbe:
          httpGet:
            path: /
            port: 13133
      volumes:
        - configMap:
            name: otel-collector-conf
            items:
              - key: otel-collector-config
                path: otel-collector-config.yaml
          name: otel-collector-config-vol
3

After two days of experimentation, I finally succeeded based on version 7.15.1 (opentelemetry otlp forwarded to apm server), and data can be written to Elasticsearch.
But I tested the 7.12.0 version again. According to the above document, I deployed the opentelemetry collector, and the data was also sent to the apm server, but the service was not found on kibana. I checked the log of the apm server and it received The message that the collector forwarded to it, but it was not stored in Elasticsearch.

Generally speaking, things have made some progress, and I am probably familiar with the series relationship between opentelemetry and elastic.

vim /etc/otel-contrib-collector/config.yaml 

receivers:
  otlp:
    protocols:
      grpc:
exporters:
  logging:
    logLevel: debug
  elastic:
    apm_server_url: http://192.168.51.134:8200
processors:
  batch:
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [elastic, logging]
      processors: [batch]

AspNetCore/appsettings.json 
{
  "Logging": {
    "LogLevel": {
      "Default": "Debug",
      "Microsoft": "Debug",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "AllowedHosts": "*",
  "UseExporter": "Otlp",
  "UseLogging": true,
  "Otlp": {
    "ServiceName": "otlp-test-aspnet-core",
  },
  "AspNetCoreInstrumentation": {
    "RecordException": "true"
  }
}

The following is the log of apm server, I can already see the information from curl.

localtesting_7.12.0_apm-server | {"log.level":"info","@timestamp":"2021-12-04T07:55:55.611Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.64.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"65591513-69f7-45d7-9df4-72877ec7b4f6","event.duration":97654,"http.response.status_code":200,"ecs.version":"1.6.0"}
localtesting_7.12.0_apm-server | {"log.level":"info","@timestamp":"2021-12-04T07:56:05.682Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.64.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"21136b88-c176-4f52-832b-12e12cccc5b4","event.duration":119043,"http.response.status_code":200,"ecs.version":"1.6.0"}
localtesting_7.12.0_apm-server | {"log.level":"info","@timestamp":"2021-12-04T07:56:11.383Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request accepted","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.11.0 go/go1.15.8","source.address":"pipe","http.request.id":"97ee33d8-8d16-4a31-93d5-032fd3ae4088","event.duration":10000515512,"http.response.status_code":202,"ecs.version":"1.6.0"}
localtesting_7.12.0_apm-server | {"log.level":"info","@timestamp":"2021-12-04T07:56:15.758Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.64.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"7c8b6dbb-02c2-44c4-acbf-bf42c9c4f137","event.duration":118150,"http.response.status_code":200,"ecs.version":"1.6.0"}
localtesting_7.12.0_apm-server | {"log.level":"info","@timestamp":"2021-12-04T07:56:25.865Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.64.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"b0e4b851-2cca-44a1-942d-27c52d64a63e","event.duration":112893,"http.response.status_code":200,"ecs.version":"1.6.0"}

image
image1209×613 59.9 KB

4

I found a lot of opentelemetry demos through debugging, and I succeeded.
There is a strange place, I need to remind everyone that the 7.12 version of apmserver can also directly receive otlp messages and write them to Elasticsearch.

5

Thanks @wajika .
Please note that the OpenTelemetry Collector for Elastic is now deprecated and replaced by the native OTLP support provided by APM Server.
The migration path is described on opentelemetry-collector-contrib/exporter/elasticexporter at main · open-telemetry/opentelemetry-collector-contrib · GitHub

The OpenTelemetry Collector configuration should look like:

...
exporters:
  otlp/elastic:
      endpoint: "xxx.elastic-cloud.com:443"
      headers:
          Authorization: "Bearer your-apm-secret-token"
service:
  pipelines:
    metrics:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - otlp/elastic
    traces:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - otlp/elastic
1 Like
6

Thank you for your reply. I hope that the elastic team can share more diverse application cases about opentelemetry+Elasticsearch, especially recently I saw that version 7.16 supports monitoring the features of CICD pipeline, but there is only an introduction to the use of jenkins on the docs. I look forward to it. See the sharing of other tools.

By the way, we use the .net development environment.

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.