Docker Compose ELK 8.10.2

Elastic Stack Elasticsearch
1

Hi,

I'm trying to create a docker-compose.yml file by following the instructions here.

The only change I made to the docker-compose.yml file was to add the Logstash service.

This is what I added to the docker-compose.yml file from the example:

logstash_test:
    depends_on:
      es01:
        condition: service_healthy
      es02:
        condition: service_healthy
      es03:
        condition: service_healthy
    image: logstash:8.10.2
    volumes:
      - "./logstash.yml:/usr/share/logstash/config/logstash.yml"
      - "./logstash.conf:/usr/share/logstash/pipeline/logstash.conf"

This is my logstash.conf:

input {

  heartbeat {
    message => "Hello"
    interval => "5"
  }
}

filter {
    
}

output {

  stdout { 
    id => "all_output"
    codec => rubydebug 
  }
  

  # Write data to ES
  elasticsearch {
    # hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]
    hosts => ["https://localhost:9200"]
    index => "emittr"
    user => "elastic"
    password => "changeme"
  }
}

This is my logstash.yml file:

config.reload.automatic: true
config.reload.interval: 5s
xpack.monitoring.enabled: false

When I run docker compose up -d all the containers start like they should but my Logstash container is not able to connect to my ES Cluster.

I'm getting the following connection refused error:

[2023-10-11T00:24:55,669][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused>}
[2023-10-11T00:24:55,670][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused"}

Not sure why I can't get Logstash to connect to my ES Cluster even though I've verified port 9200 is open.

2

Most likely It's because your are not supplying the ca cert for the self signed certs to connect to elasticsearch

Just like Kibana

Or you can set verification to none

3

Using localhost means that Logstash will try to connect to an Elasticsearch instance running on the same machine/container, which is not your case since they are on different containers.

You need to use the hostnames of your containers as you did here:

But since you are using https you will have to provide the ca certificate as @stephenb mentioned or set ssl_verification_mode to none in the elasticsearch output.

4

I changed my elastic output to this:

  elasticsearch {
    hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]
    index => "emittr"
    user => "elastic"
    password => "changeme"
    ssl_verification_mode => "none"
  }

But now getting this error:

[2023-10-11T15:50:46,488][ERROR][logstash.outputs.elasticsearch][main] Failed to install template {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://es02:9200/_index_template/ecs-logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:337:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:411:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:323:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:331:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:405:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:410:in `template_exists?'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:81:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:44:in `install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:32:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch.rb:595:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch.rb:353:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch.rb:326:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:172:in `block in after_successful_connection'"]}
[2023-10-11T15:50:47,552][ERROR][logstash.outputs.elasticsearch][main][28975a773635c65c57ead533edf58c162c7c5f2ed144b0e8cf5c9d5061958ab9] Encountered a retryable error (will retry with exponential backoff) {:code=>401, :url=>"https://es03:9200/_bulk", :content_length=>168, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}"}
5

I'm able to authenticate using elastic/changeme as the creds when I try accessing the ES cluster from the webui and get the following response:

{
  "name" : "es01",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "wwWN4YRoQmefhbfh3C4TCw",
  "version" : {
    "number" : "8.10.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "6d20dd8ce62365be9b1aca96427de4622e970e9e",
    "build_date" : "2023-09-19T08:16:24.564900370Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
6

Made a little progress using this output config:

# Write data to ES
  elasticsearch {
    hosts => ["https://es01:9200"]
    index => "emittr"
    user => "elastic"
    password => "changeme"
    ssl_verification_mode => "none"
  }

When I removed hosts es02 and es03 from the hosts list then I was able to successfully write to ES.

When those to 2 hosts were enabled in the config I was getting these authentication errors:

[2023-10-12T17:03:29,405][ERROR][logstash.outputs.elasticsearch][main][f205d11d7a0e48058903e0dfec13d594ff70b14d04a12f6edccae5953aa7ea48] Encountered a retryable error (will retry with exponential backoff) {:code=>401, :url=>"https://es02:9200/_bulk", :content_length=>168, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}"}
[2023-10-12T17:03:30,611][ERROR][logstash.outputs.elasticsearch][main][f205d11d7a0e48058903e0dfec13d594ff70b14d04a12f6edccae5953aa7ea48] Encountered a retryable error (will retry with exponential backoff) {:code=>401, :url=>"https://es03:9200/_bulk", :content_length=>168, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}"}
1 Like
7

What is the result of a curl to https://es01:9200/_cluster/health?pretty ?

8

@mohsin106 Great Catch!!

Looking closer at the compose

    ports:
      - ${ES_PORT}:9200

The 9200 port is only exposed on es01 I never noticed that before!

I have to think why that is an issue... but seems related..

I would think the are all on the local docker network... its is possible since the first uses 9200 the others use 9201 9202 etc

1 Like
9

Fwiw, I had some connection issues when I was initially setting up my docker services as well. I had ended up switching from container names to hosts => ["https://tasks.elasticsearch:9200"] which apparently uses some docker internal networking to reference the container services since I couldn't connect via <service>_app.

If you move past using docker-compose for it and making services of the stack it might be something to try if you have networking issues again.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.