Not seeing logs from OTEL Collector in APM

Elastic Observability
1

Hey folks,
My company decided to start implementing "traces\spans" into our stack and we would like to implement observability over them if possible.

I'm using this docker compose to launch a somewhat working elk stack to be able to just view a working POC of the traces being viewed (this example won't be a production grade deployment that is to say)

This is the guide i used in-order to spin everything up and upbring the current half-working stack

The steps taken are these -

1.Running docker compose to bring up the stack
2.Setting up the output of the from the "fleet" settings mentioned in the guide i linked above in this stage "Reconfigure output, add certificate"
3.After doing so, i am able to some "Kibana" derived traces and information

image
image900×315 39.2 KB

4.Doing some calls to my python application that the docker compose runs

for example


command ->

curl -vv localhost:8000/

output ->

{"message":"Failed Successfully :)"}[

Then, when checking APM board in Kibana and see my trace

image
image900×320 37.9 KB

Now comes the issue, we have set up in an nginx-alpine-otel official image, which is configured to send traces into "otel collector",
which is running in from this docker image -
otel/opentelemetry-collector-contrib:0.120.0

And has this configuration file running with it otel collector configuration file · GitHub

(you will see in the configuration file some weird conventions for specifying ports in the configuration linked, u can assume it's static ports which are working, meaning - they receive traffic)

Now I am seeing in this otel collector, logs from nginx being received
for example:

And when I look at the elastic agent docker container logs, I do see the logs being received from the collector,

But again In the UI I do not see a service_named “Frontend-Gateway-linux-qa”
Like the span is sending, what am I missing ?

Would love and appreciate advise on this!

Thanks in advance guys

2 
  otlphttp/kibana:
    traces_endpoint: "http://10.0.1.158:8200"

should be

  otlphttp/kibana:
    traces_endpoint: "http://10.0.1.158:8200/v1/traces"

or

  otlphttp/kibana:
    endpoint: "http://10.0.1.158:8200"

When using endpoint the exporter automatically appends /v1/traces for traces.

The APM Server has a root API node which, unfortunately for this case, returns a 200 success when you POST against it:

curl -X POST http://127.0.0.1:8200/ \
  -H "Authorization: Bearer secret_token"

{
  "build_date": "2021-12-18T19:59:06Z",
  "build_sha": "24fe620eeff5a19e2133c940c7e5ce1ceddb1445",
  "publish_ready": true,
  "version": "8.17.2"
}

So I think what's happening is that the OTLP HTTP Exporter is getting a 200 response code when it POSTS the batch, which it thinks means the batch succeeded. So you get no feedback that something isn't working, and the traces never land in Elasticsearch.

3

Thank you so much sir!
it works! , appreciated so much

1 Like
4

Opened an enhancement request: Return an error on POST requests to `/` · Issue #15965 · elastic/apm-server · GitHub

2 Likes