# Docker Compose ELK 8.10.2

**URL:** <https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [October 11, 2023, 12:33am UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770 "2023-10-11T00:33:50Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [October 11, 2023, 12:33am UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/1 "2023-10-11T00:33:50Z")

</div>

Hi,

I'm trying to create a docker-compose.yml file by following the instructions [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-compose-file).

The only change I made to the docker-compose.yml file was to add the Logstash service.

This is what I added to the docker-compose.yml file from the [example](https://github.com/elastic/elasticsearch/blob/8.10/docs/reference/setup/install/docker/docker-compose.yml):

```auto
logstash_test:
    depends_on:
      es01:
        condition: service_healthy
      es02:
        condition: service_healthy
      es03:
        condition: service_healthy
    image: logstash:8.10.2
    volumes:
      - "./logstash.yml:/usr/share/logstash/config/logstash.yml"
      - "./logstash.conf:/usr/share/logstash/pipeline/logstash.conf"

```

This is my logstash.conf:

```auto
input {

  heartbeat {
    message => "Hello"
    interval => "5"
  }
}

filter {
    
}

output {

  stdout { 
    id => "all_output"
    codec => rubydebug 
  }
  

  # Write data to ES
  elasticsearch {
    # hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]
    hosts => ["https://localhost:9200"]
    index => "emittr"
    user => "elastic"
    password => "changeme"
  }
}

```

This is my logstash.yml file:

```auto
config.reload.automatic: true
config.reload.interval: 5s
xpack.monitoring.enabled: false

```

When I run `docker compose up -d` all the containers start like they should but my Logstash container is not able to connect to my ES Cluster.

I'm getting the following connection refused error:

```auto
[2023-10-11T00:24:55,669][INFO][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused>}
[2023-10-11T00:24:55,670][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused"}

```

Not sure why I can't get Logstash to connect to my ES Cluster even though I've verified port 9200 is open.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 11, 2023, 5:30am UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/2 "2023-10-11T05:30:43Z")

</div>

Most likely It's because your are not supplying the ca cert for the self signed certs to connect to elasticsearch

> **[Elasticsearch output plugin | Logstash Reference \[8.10\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ssl_certificate_authorities)**

Just like Kibana

Or you can set verification to none

> **[Elasticsearch output plugin | Logstash Reference \[8.10\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ssl_verification_mode)**

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 11, 2023, 11:15am UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/3 "2023-10-11T11:15:00Z")

</div>

> [@mohsin106](#):
>
> `hosts => ["https://localhost:9200"]`

Using `localhost` means that Logstash will try to connect to an Elasticsearch instance running on the **same** machine/container, which is not your case since they are on different containers.

You need to use the hostnames of your containers as you did here:

> [@mohsin106](#):
>
> `hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]`

But since you are using `https` you will have to provide the ca certificate as @stephenb mentioned or set `ssl_verification_mode` to `none` in the elasticsearch output.

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [October 11, 2023, 3:53pm UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/4 "2023-10-11T15:53:12Z")

</div>

I changed my elastic output to this:

```auto
  elasticsearch {
    hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]
    index => "emittr"
    user => "elastic"
    password => "changeme"
    ssl_verification_mode => "none"
  }

```

But now getting this error:

```auto
[2023-10-11T15:50:46,488][ERROR][logstash.outputs.elasticsearch][main] Failed to install template {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://es02:9200/_index_template/ecs-logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:337:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:411:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:323:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:331:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:405:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:410:in `template_exists?'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:81:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:44:in `install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:32:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch.rb:595:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch.rb:353:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch.rb:326:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:172:in `block in after_successful_connection'"]}
[2023-10-11T15:50:47,552][ERROR][logstash.outputs.elasticsearch][main][28975a773635c65c57ead533edf58c162c7c5f2ed144b0e8cf5c9d5061958ab9] Encountered a retryable error (will retry with exponential backoff) {:code=>401, :url=>"https://es03:9200/_bulk", :content_length=>168, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}"}

```

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [October 11, 2023, 3:56pm UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/5 "2023-10-11T15:56:03Z")

</div>

I'm able to authenticate using `elastic/changeme` as the creds when I try accessing the ES cluster from the webui and get the following response:

```auto
{
  "name" : "es01",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "wwWN4YRoQmefhbfh3C4TCw",
  "version" : {
    "number" : "8.10.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "6d20dd8ce62365be9b1aca96427de4622e970e9e",
    "build_date" : "2023-09-19T08:16:24.564900370Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

```

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [October 12, 2023, 5:09pm UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/6 "2023-10-12T17:09:05Z")

</div>

Made a little progress using this output config:

```auto
# Write data to ES
  elasticsearch {
    hosts => ["https://es01:9200"]
    index => "emittr"
    user => "elastic"
    password => "changeme"
    ssl_verification_mode => "none"
  }

```

When I removed hosts `es02` and `es03` from the `hosts` list then I was able to successfully write to ES.

When those to 2 hosts were enabled in the config I was getting these authentication errors:

```auto
[2023-10-12T17:03:29,405][ERROR][logstash.outputs.elasticsearch][main][f205d11d7a0e48058903e0dfec13d594ff70b14d04a12f6edccae5953aa7ea48] Encountered a retryable error (will retry with exponential backoff) {:code=>401, :url=>"https://es02:9200/_bulk", :content_length=>168, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}"}
[2023-10-12T17:03:30,611][ERROR][logstash.outputs.elasticsearch][main][f205d11d7a0e48058903e0dfec13d594ff70b14d04a12f6edccae5953aa7ea48] Encountered a retryable error (will retry with exponential backoff) {:code=>401, :url=>"https://es03:9200/_bulk", :content_length=>168, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}"}

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 12, 2023, 5:20pm UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/7 "2023-10-12T17:20:13Z")

</div>

What is the result of a curl to `https://es01:9200/_cluster/health?pretty` ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 12, 2023, 5:32pm UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/8 "2023-10-12T17:32:48Z")

</div>

@mohsin106 Great Catch!!

Looking closer at the compose

```auto
    ports:
      - ${ES_PORT}:9200

```

The 9200 port is only exposed on `es01` I never noticed that before!

I have to think why that is an issue... but seems related..

I would think the are all on the local docker network... its is possible since the first uses 9200 the others use 9201 9202 etc

---

<div class="post-metadata">

**Author:** ![Matt\_Clairmont](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_clairmont/32/125860_2.png) [@Matt\_Clairmont](https://discuss.elastic.co/u/Matt_Clairmont)\
**Post date:** [October 13, 2023, 2:07pm UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/9 "2023-10-13T14:07:41Z")

</div>

Fwiw, I had some connection issues when I was initially setting up my docker services as well. I had ended up switching from container names to `hosts => ["https://tasks.elasticsearch:9200"]` which apparently uses some docker internal networking to reference the container services since I couldn't connect via `<service>_app`.

If you move past using docker-compose for it and making services of the stack it might be something to try if you have networking issues again.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 10, 2023, 2:08pm UTC](https://discuss.elastic.co/t/docker-compose-elk-8-10-2/344770/10 "2023-11-10T14:08:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.