Issue with otlp-collectors and elastic

Elastic Stack Elasticsearch
1

Hi, It looks like smth block search/view of metric data in real-time mode (from otel-collectors) and produces this data incrementally. For example, on 12 June I could check data till 01:00, but at 13 June I could see data till 07:00 12 June.

On dashboards side:

image
image2554×1129 343 KB

I see this on otlp-collectors log:

image
image1920×686 194 KB

2

Thanks for reaching out, @iamp3. Please provide some further context as to the issue you are facing and what you expect to see versus what you are seeing. Also, since screenshots can be hard to read, could you paste in the errors you are receiving?

3

Sorry for late response, logs:

Value: 0
    {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2024-08-23T09:21:28.238Z    info    exporterhelper/retry_sender.go:129    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "53.841742873s"}
2024-08-23T09:21:30.219Z    info    exporterhelper/retry_sender.go:129    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "1m9.974494263s"}
2024-08-23T09:21:45.116Z    info    exporterhelper/retry_sender.go:129    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "43.117012577s"}
2024-08-23T09:21:46.499Z    error    exporterhelper/queue_sender.go:125    Exporting failed. No more retries left. Dropping data.    {"kind": "exporter", "data_type": "metrics", "name": "otlp/elastic", "error": "max elapsed time expired rpc error: code = DeadlineExceeded desc = context deadline exceeded", "dropped_items": 209}

We expect to see metrics/traces on elk side, but looks liike most of our data dropped with the error above

4

Thanks for following up, @iamp3.

Here are a few ideas to consider:

  • You will want to ensure that you have configured both exporters (gRPC and HTTP) to use different ports
    Double-check that you are correctly configuring the OTLP exporter and that your network is configured correctly to connect to the Elastic OTLP endpoint.
  • Here is an example of a collector I recently used, which could be helpful:

Do you have the YAML configuration file you can send over? You may also want to check out our documentation on the subject.