Opentelemetry nodejs: cannot set language nor service.environment

[ccontini (Clément Contini)]

Kibana version: 7.14.1

Elasticsearch version: 7.14.1

APM Server version: 7.14.1

Open-telemetry collector version: 0.36

Description of the problem including expected versus actual behavior:

I am using opentelemetry to send traces to Elastic APM. My problem is that my nodejs application is not recognized as such (it shows the generic opentelemetry logo) and I am not able to set the deployment environment either.

I have instrumented another application in java using the opentelemetry java agent, and this one shows up properly as a java application and has the proper environment.

In my java application, I see this in the trace metadata:

But in my nodejs application, they appear under labels:

This is how I declare the attributes for my tracer:

    const provider = new NodeTracerProvider({
        resource: new Resource({
            [ResourceAttributesSC.SERVICE_NAME]: serviceName,
            [ResourceAttributesSC.SERVICE_NAMESPACE]: "API",
            [ResourceAttributesSC.DEPLOYMENT_ENVIRONMENT]: "dev",
            [ResourceAttributesSC.TELEMETRY_SDK_NAME]: "opentelemetry/nodejs",
            [ResourceAttributesSC.TELEMETRY_SDK_LANGUAGE]: "nodejs",
        }),
    });

Note that I do see the serviceName with the proper value in the UI.

I was not able to configure my tracer in such a way that these metadata are interpreted properly by APM.

[riferrei (Ricardo Ferreira)]

Hi @ccontini,

I'm not an expert in Node.JS, but I had noticed the same behavior when I was instrumenting a Go application. Then, I upgraded the client libraries from OpenTelemetry and got it working. In your case, I can see that you're using ResourceAttributesSC to specify the language. In a quick peek in the latest release of the Node.JS client for OTel — I can see that the required classes to specify have changed:

open-telemetry/opentelemetry-js/blob/3045eba229f031e1510785dbb9713f24daf17ed1/packages/opentelemetry-resources/src/Resource.ts#L42

My recommendation is to upgrade your client libraries and the APM Server, whose current version is 7.15.1. See if it fixes the problem. If it doesn't, we can try file an issue on the APM Server project on GitHub, together

— @riferrei

[ccontini (Clément Contini)]

Hello @riferrei,

Thanks for your answer.

Regarding the attributes, I am actually importing the attributes like this:

const { SemanticResourceAttributes: ResourceAttributesSC } = require('@opentelemetry/semantic-conventions');

So I believe this part is correct.

I have updated the APM server to 7.15.1 but nothing changed.

I wanted to mention that the spans I generate from otel look like this:

{
   "resourceSpans":[
      {
         "resource":{
            "attributes":[
               {
                  "key":"service.name",
                  "value":{
                     "stringValue":"myapp"
                  }
               }
            ]
         },
         "instrumentationLibrarySpans":[
            {
               "instrumentationLibrary":{
                  "name":"@opentelemetry/instrumentation-pg",
                  "version":"0.25.0"
               },
               "spans":[
                  {
                     "traceId":"ea3c106f996566c04038c066e2bb4077",
                     "spanId":"63c9cdc65f45ae31",
                     "parentSpanId":"cedcde76131ed10d",
                     "name":"pg.query:SET",
                     "kind":"SPAN_KIND_CLIENT",
                     "startTimeUnixNano":"1634845150742047000",
                     "endTimeUnixNano":"1634845150743406000",
                     "attributes":[
                        {
                           "key":"db.name",
                           "value":{
                              "stringValue":"dev"
                           }
                        },
                        {
                           "key":"db.system",
                           "value":{
                              "stringValue":"postgresql"
                           }
                        },
                        {
                           "key":"db.connection_string",
                           "value":{
                              "stringValue":"jdbc:postgresql://localhost:5432/dev"
                           }
                        },
                        {
                           "key":"net.peer.name",
                           "value":{
                              "stringValue":"localhost"
                           }
                        },
                        {
                           "key":"net.peer.port",
                           "value":{
                              "doubleValue":5432
                           }
                        },
                        {
                           "key":"db.user",
                           "value":{
                              "stringValue":"postgres"
                           }
                        },
                        {
                           "key":"db.statement",
                           "value":{
                              "stringValue":"SET client_min_messages TO warning;SET TIME ZONE INTERVAL '+00:00' HOUR TO MINUTE;"
                           }
                        },
                        {
                           "key":"perlin",
                           "value":{
                              "stringValue":"pinpin"
                           }
                        },
                        {
                           "key":"service.name",
                           "value":{
                              "stringValue":"myapp"
                           }
                        },
                        {
                           "key":"telemetry.sdk.language",
                           "value":{
                              "stringValue":"nodejs"
                           }
                        },
                        {
                           "key":"telemetry.sdk.name",
                           "value":{
                              "stringValue":"opentelemetry/nodejs"
                           }
                        },
                        {
                           "key":"telemetry.sdk.version",
                           "value":{
                              "stringValue":"1.0.0"
                           }
                        },
                        {
                           "key":"service.namespace",
                           "value":{
                              "stringValue":"API"
                           }
                        },
                        {
                           "key":"deployment.environment",
                           "value":{
                              "stringValue":"dev"
                           }
                        },
                        {
                           "key":"service.language.name",
                           "value":{
                              "stringValue":"nodejs"
                           }
                        }
                     ],
                     "status":{
                        
                     }
                  }
               ]
            }
         ]
      }
   ]
}

And the only attribute that is read properly is the service.name under resourceSpans.resource.attributes. Everything else is under resourceSpans. instrumentationLibrarySpans.spans.attributes and appear under labels un the UI. (I have the same problem with spans for http requests, where the status code gets added under labels.http_status_code for example, so I don't know the status of the request from the UI unless I go check the labels.

[alanstorm]

@ccontini Thanks for reporting and for your patience.

Would you be able to share the version of the OpenTelemetry packages you're using for Node.js? I'm asking because when I use the latest and greatest from OpenTelemetry

  "dependencies": {
    "@opentelemetry/api": "^1.0.0",
    "@opentelemetry/sdk-trace-node": "^1.0.0",
    "@opentelemetry/sdk-trace-base": "^1.0.0",
    "@opentelemetry/exporter-otlp-grpc": "^0.26.0",
    "@opentelemetry/exporter-otlp-http": "^0.26.0",
    "@opentelemetry/instrumentation-http": "^0.26.0",
    "express": "^4.17.1",
    "nodemon": "^2.0.14"
  }

and a bootstrap that looks like this

const { diag, DiagConsoleLogger, DiagLogLevel } = require("@opentelemetry/api");
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const { ConsoleSpanExporter, SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
const { OTLPTraceExporter } =  require('@opentelemetry/exporter-otlp-grpc');
const core = require("@opentelemetry/core");
const fs = require('fs')

const grpc = require('@grpc/grpc-js');
process.env['OTEL_EXPORTER_OTLP_HEADERS'] = 'Authorization=Bearer D...a'
const baseUrl = 'https://[your-server].apm.us-east-1.aws.cloud.es.io:443'
const collectorOptions = {
  url: baseUrl,
  credentials: grpc.credentials.createSsl(),
};

const provider = new NodeTracerProvider();

diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ALL);

provider.addSpanProcessor(
  new SimpleSpanProcessor(
    new OTLPTraceExporter(collectorOptions)
  )
);

provider.register();

registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
  ],
});

My spans automatically have the correct resource attributes added to them

    "resource": {
      "attributes": {
        "service.name": "unknown_service:node",
        "telemetry.sdk.language": "nodejs",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.0.0"
      }
    },

and the Node.js icon appears in my service list, (i.e. identifying the service correctly)

If possible, I'd recommend you update to the 1.x/0.26.x packages -- a lot's been changing on the Node.js side recently.

If that's not possible, would you be able to share the versions of the OpenTelemetry package (or metapackage version if someone else is packaging this for you) you're using? With that information we can might be able to reproduce your problem, and if we reproduce your problem we can usually find a solution.

[ccontini (Clément Contini)]

Hello @alanstorm,

Thank you for your suggestion, I was finally able to make it work properly!

The thing is I was already using the latest versions for all of these packages, however I was using the Jaeger exporter and not exporter-otlp-grpc. Switching exporter worked perfectly, the platform is recognized as node and my environment is tagged. I also see the status_code properly under http.response.status_code instead of labels.

My last question is: is it normal that the Jaeger exporter did not work as I expected? Is it a misconfiguration on my part or the setup is just not supported? To be clear, I was doing node -> jaeger -> otel-collector -> apm -> Elasticsearch.

Thanks again!

[axw (Andrew Wilkins)]

@ccontini

My last question is: is it normal that the Jaeger exporter did not work as I expected? Is it a misconfiguration on my part or the setup is just not supported? To be clear, I was doing node -> jaeger -> otel-collector -> apm -> Elasticsearch.

We do support receiving data from Jaeger, but we expect these attributes (telemetry.sdk.language and friends) to then come in as "process tags", Jaeger's equivalent of OTel resource attributes. It would seem that the Jaeger exporter is not doing that.

I think you will have a much better time in general if you use the OTLP exporter, as there will be fewer points of translation and thus fewer opportunities for things to get mistranslated between the data models.

[alanstorm]

Glad you got it working @ccontini!

Regarding

is it normal that the Jaeger exporter did not work as I expected? Is it a misconfiguration on my part or the setup is just not supported? To be clear, I was doing node -> jaeger -> otel-collector -> apm -> Elasticsearch.

I wouldn't say the behavior is or isn't expected -- but Elastic strongly recommends folks use the OTLP endpoint built into APM Server as an OpenTelemetry collector and ship data to it using an OTLP exporter from their language (which is what I describe above). Using this method reduces the number of moving parts in your system and is going to be the easiest to support going forward.

The challenge with your setup is

1. Node sends the data to jaeger, presumably via the @opentelemetry/exporter-jaeger exporter, and this exporter might alter the data

2. Jaeger receives the data, and has a change to alter it via unknown Jaeger processes

3. Jaeger sends the data to an OpenTelemetry collector (how? also, this is another thing that might alter the data)

4. The collector exports data to APM Server (how? also, this is another thing that might alter the data)

Basically, the more round trips data makes the greater a chance something is going to transform the data in an unexpected way.

@ccontini in addition to the how? questions above, I have one more: Is there some advantage or feature the node -> jaeger -> otel-collector -> apm -> Elasticsearch setup gives you, or was this just the path you got working?

[ccontini (Clément Contini)]

Thank you @axw and @alanstorm for your answers.

@axw you are right I did not configure anything on the jaeger side to do that, but when I was adding these attributes in the otel-collector (after jaeger in the pipeline) processor, these attributes were still saved under labels.

@axw @alanstorm I actually did not really intended to keep jaeger for my final setup, I was just trying it out (I am still new with tracing, and I am exploring different things). I completely agree there is no real advantage in my setup running the jaeger before otel-collector, and will stick to otel-collector for the future.

Thanks!