Unable to send data from Otel Collector to Elasticsearch using Elastic APM

Elastic Stack Elasticsearch
1

Hi Team,
I am new to OpenTelemetry. I am trying to do setup using yaml files appraoch but facing 'connection refused' issue while connecting apm to elasticsearch.

What I have achieved so far:

  1. Able to receive data in the OtelCollector.
  2. Able to receive data in the Prometheus.

Issues facing:

  1. Not able to connect APM with Elasticsearch.
    Error
    Error1655×69 5.26 KB

Stack:
Otel Collector
Elastic APM
Elasticsearch
Prometheus

I am attaching all yml files for reference. Please help me in resolving issues and point out other mistakes I am doing right now.

docker-compose.yaml

version: "3.7"
services:     
# APM Server   
  apm-server:
    image: docker.elastic.co/apm/apm-server:8.8.2
    volumes:
      - ./apm-server.docker.yml:/usr/share/apm-server/apm-server.yml
    cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
    cap_drop: ["ALL"]
    ports:
      - "0.0.0.0:8200:8200"
    environment:
      host: 0.0.0.0:8200
    depends_on:
      - elasticsearch
    command: >
       apm-server -e
         -E apm-server.rum.enabled=true
         -E output.elasticsearch.hosts=['localhost:9200']
    networks:
      - icmnetworks  
# Elasticsearch     
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.8.2
    container_name: elasticsearch
    volumes:
      - ./elasticsearch.yaml:/usr/share/elasticsearch/config/elasticsearch.yml
    #volumes:
      #- /usr/share/elasticsearch/data
      #- C:\ElasticSearch\elasticsearch-7.17.0\data:/usr/share/elasticsearch/data
    ports:
      - "0.0.0.0:9200:9200"
      - "0.0.0.0:9300:9300"
    environment:
      transport.host: 127.0.0.1
      ES_JAVA_OPTS: -Xms512m -Xmx512m
    mem_limit: 1073741824
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - icmnetworks      
  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - 9090:9090
    networks:
      - icmnetworks  
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./output:/etc/output:rw
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - 8888:8888   # Prometheus metrics exposed by the collector
      - 8889:8889   # Prometheus exporter metrics
      - 4317:4317   # OTLP gRPC receiver
      - 4318:4318   # OTLP http receiver
    depends_on:
      - prometheus
      - elasticsearch
    networks:
      - icmnetworks

networks:
  icmnetworks:
    driver: bridge

otel-collector-config.yaml

receivers:
    otlp:
        protocols:
            grpc:
            http:

processors:
    batch:
  
exporters:
    file:
        path: /etc/output/logs.json
    prometheus:
        endpoint: "0.0.0.0:9090"
    otlp/elastic:
        endpoint: "apm-server:8200"
        tls:
            insecure: true
    logging:
        loglevel: debug

service:
    pipelines:
        traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [logging, otlp/elastic]
        metrics:
            receivers: [otlp]
            processors: [batch]
            exporters: [logging, prometheus]
        logs:
            receivers: [otlp]
            processors: []
            exporters: [logging, otlp/elastic] #, file

apm-server.yaml

apm-server:
  host: "0.0.0.0:8200"
output.elasticsearch:
  hosts: ["localhost:9200", "0.0.0.0:9200"]
path.data: /usr/share/apm-server/data
path.logs: /var/log
logging.level: info
logging.to_syslog: true
logging.metrics.enabled: true
logging.files:
  path: /var/log/apm-server
  name: apm-server
  rotateeverybytes: 10485760 # = 10MB
  keepfiles: 7
  permissions: 0600
  interval: 0

elastisearch.yaml

cluster.name: elasticsearchcluster
node.name: elasticnode
node.attr.rack: r1
path.data: /usr/share/elasticsearch/data
path.logs: /usr/share/elasticsearch/logs
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node
xpack.security.enabled: false
action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*,+u*,+o*
cluster.routing.allocation.disk.threshold_enabled: false
cluster.routing.allocation.disk.watermark.low: 700mb
cluster.routing.allocation.disk.watermark.high: 600mb
cluster.routing.allocation.disk.watermark.flood_stage: 500mb
cluster.info.update.interval: 1m

prometheus.yaml

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090']
2

Hi @Prateek_Kumar,

Welcome to the community! As an FYI sharing error messages and yaml configuration is best done as code snippets or test rather than images to make it easier to read and copy.

I see in your otel-collector-config.yaml that you have tls.insecure set to true. As of Elasticsearch v8.x security is enabled by default so I wonder if that could be your issue. Can you try changing this setting and perhaps configuring the certificate?

3

Hi @carly.richmond ,
As suggessted, I updated the question.

I have aready disabled the security in Elasticsearch.yaml. I removed 'tls.insecure' from the configuration but still faced the same issue.

4

We really don't recommend disabling the security. If the endpoint details are correct I think the problem might be the lack of a specified API key. Can you register and specify a API key similar to this example in the documentation and see if that resolves the problem?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.