Logstash not connecting to Elasticsearch - using Docker-Compose

Both containers are running and are attempting to communicate with each other. I see Filebeat connecting to Logstash and send information, however I don't see any data being read into Elasticsearch.

Here are the logs that pop up when I ran the logstash container:

[2022-01-26T18:55:14,153][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2022-01-26T18:55:14,420][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2022-01-26T18:55:14,432][INFO ][logstash.licensechecker.licensereader] Elasticsearch version determined (7.16.2) {:es_version=>7}
[2022-01-26T18:55:14,434][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-01-26T18:55:14,530][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2022-01-26T18:55:14,530][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2022-01-26T18:55:14,642][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-01-26T18:55:15,332][INFO ][org.reflections.Reflections] Reflections took 66 ms to scan 1 urls, producing 119 keys and 417 values 
[2022-01-26T18:55:16,385][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2022-01-26T18:55:16,385][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2022-01-26T18:55:16,393][WARN ][deprecation.logstash.inputs.beats] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2022-01-26T18:55:16,410][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2022-01-26T18:55:16,426][WARN ][deprecation.logstash.outputs.elasticsearchmonitoring] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2022-01-26T18:55:16,478][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elasticsearch:9200"]}
[2022-01-26T18:55:16,496][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2022-01-26T18:55:16,517][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2022-01-26T18:55:16,522][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (7.16.2) {:es_version=>7}
[2022-01-26T18:55:16,522][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-01-26T18:55:16,589][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)

For reference, here is my docker-compose.yml file:

version: '2.2' 
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
    container_name: elasticsearch
    environment:
      - node.name=es01
      - discovery.type=single-node
      - cluster.name=es-docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - data01:/usr/share/elasticsearch/data
    ports:
      - xxxx:9200
  kibana:
    image: docker.elastic.co/kibana/kibana:7.16.2
    container_name: kibana
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
      ELASTICSEARCH_HOSTS: '["http://elasticsearch:9200"]'
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    ports:
      - xxxx:5601

  logstash:
    image: docker.elastic.co/logstash/logstash:7.16.2
    container_name: logstash
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
      ELASTICSEARCH_HOSTS: '["http://elasticsearch:9200"]'
    volumes:
      - ./logstash/config/logstashconfig.config:/usr/share/logstash/logstashC.config
    depends_on:
      - elasticsearch
    ports:
      - xxxx:5044
volumes:
  data01:
    driver: local

Can you share your logstash configuration?

From your logstash logs there is no issue for the logstash container to talk with the Elasticsearch container.

[2022-01-26T18:55:14,153][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2022-01-26T18:55:14,420][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2022-01-26T18:55:14,432][INFO ][logstash.licensechecker.licensereader] Elasticsearch version determined (7.16.2) {:es_version=>7}

Those line, and others, means that your logstash talked with your Elasticsearch without any issues.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.