Hello everyone,
I am trying to send traces directly from Open Telemetry Collector to an Elasticsearch container (I do not want to use Elastic APM). My current problem is that the traces are reaching the Elasticsearch container, but generate the following error:
{"@timestamp":"2024-04-11T23:08:02.254Z", "log.level": "WARN", "message":"http channel [Netty4HttpChannel{localAddress=/173.19.0.3:9200, remoteAddress=/173.19.0.1:34642}] missing tracking channel", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[node-single][transport_worker][T#4]","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"uid","elasticsearch.node.id":"id","elasticsearch.node.name":"node-single","elasticsearch.cluster.name":"docker-ELK"}
It appears that my OpenTelemetry Collector or Elasticsearch config file is missing a configuration, but I cannot pinpoint the error.
Here is my OTEL Collector yaml:
receivers:
otlp:
protocols:
grpc:
http:
exporters:
logging:
loglevel: debug
elasticsearch/trace:
endpoints: [http://elasticsearch:9200]
index: "otel"
user: elastic
password:
api_key:
tls:
insecure_skip_verify: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [elasticsearch/trace]
And here is my Elasticsearch yaml:
node.name: node-single
path.data: /usr/share/elasticsearch/data
path.repo: ["/es-data/snapshots/"]
path.logs: /var/log
cluster.name: docker-ELK
network.host: 0.0.0.0
thread_pool.search.queue_size: 100000
xpack.security.enabled=false
#xpack.monitoring.collection.enabled: true
xpack.monitoring.collection.interval: 30s # default 10s
xpack.monitoring.history.duration: 3d #In requirements this should be 15d (?)
xpack.monitoring.exporters.my_local:
type: local
use_ingest: false
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/elasticsearch/elasticsearch.key
xpack.security.http.ssl.certificate: certs/elasticsearch/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt
xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: certs/elasticsearch/elasticsearch.key
xpack.security.transport.ssl.certificate: certs/elasticsearch/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt
xpack.security.transport.ssl.verification_mode: certificate
xpack.license.self_generated.type: basic
In my application, I am sending traces via an SDK in the following manner:
var tracerProvider = Sdk.CreateTracerProviderBuilder()
.AddSource(MyActivitySource.Name)
.AddAspNetCoreInstrumentation()
.AddOtlpExporter(exporterOptions =>
{
exporterOptions.Endpoint = new Uri("http://localhost:9200");
exporterOptions.Protocol = OtlpExportProtocol.Grpc;
})
.Build();
I would really appreciate if someone could indicate what could possibly be wrong. I am also receiving data from Metric and File Beats, could this be interfering in any way?