I have a nestjs project and i need to trace graphql calls. i am usnig opentelemetry to collect traces and apm server to show them on kibana. The problem i am facing is that all graphql calls are being grouped under same name of 'POST /graphql'. What could be the cause of this problem? secondly how can i show span.attributes of otel span on kibana? does apm server assigns a new id to each transaction or uses the span.id of span thats being exported from otel?
Hi @Hamad, the spans are being named POST /graphql because all graphql requests go to the same endpoint with the query described in the body of the request. You are using the OTel auto instrumentation for GraphQL and they have this issue, which seems to match yours. I would suggest commenting on it or following the issue to get it resolved. It seems that the instrumentation is just using the HTTP info as the span name and not doing any special handling to name the span using the query.
Regarding your second question, Elastic APM has a concept of a transaction, which is a "root span", while OTel only has a span representation. The APM server will infer when an OTel span is a "root span" and use the span id of the OTel span as the transaction id. So if you're looking for a particular OTel span, you could try finding it with the id as the span.id or transaction.id.
I hope that helps,
Emily
I am unable to find spans with id. I am printing all spans on console while exporting them but i cant find those span by id on kiabana. does this have to do something with index patterns? sorry for these dumb question, I am new to ELK
Hi @Hamad,
Could you share your APM Server logs, please?
There is nothing in logs on kibana, however i can see traces and metrics.
Some logs that are being printed on console where apm server is running are following:
{"log.level":"info","@timestamp":"2023-09-18T16:43:37.930+0500","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=tools-service-template&service.environment=development","http.request.method":"GET","user_agent.original":"apm-agent-nodejs/3.49.1 (tools-service-template 1.1.0)","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"4e702237-8c11-44fe-a7aa-cfa8ebf64c78","event.duration":331209,"http.response.status_code":304,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-18T16:43:47.754+0500","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.49.1 (tools-service-template 1.1.0)","source.address":"127.0.0.1","http.request.id":"63dd3149-449c-495a-8da8-aee795b889e1","event.duration":9992474958,"http.response.status_code":202,"ecs.version":"1.6.0"}
Hey @Hamad,
The two lines you shared are showing HTTP requests (for an application using the APM NodeJS agent apm-agent-nodejs/3.49.1, which doesn't use OTLP) reaching the APM Server. Would you mind uploading the entire APM Server log file for the time period where the OTLP requests were issued? First, I'd like to verify that the OTLP are reaching the APM Server.
The two log lines you shared aren't very useful and aren't relevant to be able to assist you.
The issue of all GraphQL calls being grouped under the same name 'POST /graphql' in your traces might be due to missing instrumentation or configuration for more granular tracing within your NestJS project. To resolve this, you should ensure that each GraphQL operation is correctly instrumented and assigned a distinct name or ID based on the operation being executed.
To display span.attributes of OpenTelemetry spans in Kibana through APM server, you'll need to configure your instrumentation to include the relevant attributes you want to capture. This often involves using custom instrumentation code to add specific attributes to your spans.
APM servers typically use the span.id from OpenTelemetry or other tracing systems to correlate spans within a transaction. The span.id is a unique identifier for each span and is used to establish relationships and dependencies between different spans in a transaction. APM servers usually do not assign new IDs but rely on the ones provided by OpenTelemetry or the tracing library in use.
Ensure that your OpenTelemetry instrumentation is correctly set up to provide distinct names or IDs for GraphQL operations and includes the necessary attributes for proper tracing and visibility in Kibana. AC Football Cases.