Not able to connect docker logstash to docker elasticsearch using basic-auth using docker-compose

Elastic Stack Logstash
1

Hi team, facing the following issue-:

While running the following docker-compose.yml, rest of the containers apart from logstash container are up and running , logstash-container fails and is exited, while trying to connect to elasticsearch logstash throws the following error saying basic-auth credentials missing, though i am providing Elasticsearch hosts, username, password in the dockerfile which i am using to construct the image of logstash.

here is the error snapshot from logs, from logstash-docker container

024-11-02 17:44:41 [2024-11-02T12:14:41,821][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-11-02 17:44:42 [2024-11-02T12:14:42,072][WARN ][logstash.licensechecker.licensereader] Health check failed {:code=>401, :url=>http://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
2024-11-02 17:44:42 [2024-11-02T12:14:42,098][WARN ][logstash.licensechecker.licensereader] Elasticsearch main endpoint returns 401 {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}"}
2024-11-02 17:44:42 [2024-11-02T12:14:42,101][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"Could not read Elasticsearch. Please check the credentials", :exception=>LogStash::ConfigurationError}
2024-11-02 17:44:42 [2024-11-02T12:14:42,286][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-11-02 17:44:42 [2024-11-02T12:14:42,463][WARN ][logstash.licensechecker.licensereader] Health check failed {:code=>401, :url=>http://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
2024-11-02 17:44:42 [2024-11-02T12:14:42,508][WARN ][logstash.licensechecker.licensereader] Elasticsearch main endpoint returns 401 {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}"}
2024-11-02 17:44:42 [2024-11-02T12:14:42,509][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Could not read Elasticsearch. Please check the credentials"}
2024-11-02 17:44:42 [2024-11-02T12:14:42,516][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
2024-11-02 17:44:42 [2024-11-02T12:14:42,683][INFO ][logstash.configmanagement.elasticsearchsource] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}}
2024-11-02 17:44:42 [2024-11-02T12:14:42,712][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2024-11-02 17:44:42 [2024-11-02T12:14:42,744][WARN ][logstash.configmanagement.elasticsearchsource] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch:9200/"}
2024-11-02 17:44:42 [2024-11-02T12:14:42,745][INFO ][logstash.configmanagement.elasticsearchsource] Elasticsearch version determined (8.15.0) {:es_version=>8}
2024-11-02 17:44:42 [2024-11-02T12:14:42,746][WARN ][logstash.configmanagement.elasticsearchsource] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
2024-11-02 17:44:42 [2024-11-02T12:14:42,848][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:44:47 [2024-11-02T12:14:47,928][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:44:52 [2024-11-02T12:14:52,905][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:44:57 [2024-11-02T12:14:57,884][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:02 [2024-11-02T12:15:02,880][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:07 [2024-11-02T12:15:07,910][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:12 [2024-11-02T12:15:12,607][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-11-02 17:45:12 [2024-11-02T12:15:12,628][WARN ][logstash.licensechecker.licensereader] Health check failed {:code=>401, :url=>http://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
2024-11-02 17:45:12 [2024-11-02T12:15:12,650][WARN ][logstash.licensechecker.licensereader] Elasticsearch main endpoint returns 401 {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}"}
2024-11-02 17:45:12 [2024-11-02T12:15:12,652][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"Could not read Elasticsearch. Please check the credentials", :exception=>LogStash::ConfigurationError}
2024-11-02 17:45:12 [2024-11-02T12:15:12,708][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-11-02 17:45:12 [2024-11-02T12:15:12,800][WARN ][logstash.licensechecker.licensereader] Health check failed {:code=>401, :url=>http://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
2024-11-02 17:45:12 [2024-11-02T12:15:12,819][WARN ][logstash.licensechecker.licensereader] Elasticsearch main endpoint returns 401 {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}"}
2024-11-02 17:45:12 [2024-11-02T12:15:12,821][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Could not read Elasticsearch. Please check the credentials"}
2024-11-02 17:45:12 [2024-11-02T12:15:12,884][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:17 [2024-11-02T12:15:17,905][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:22 [2024-11-02T12:15:22,885][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:27 [2024-11-02T12:15:27,923][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:32 [2024-11-02T12:15:32,881][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:37 [2024-11-02T12:15:37,915][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:42 [2024-11-02T12:15:42,614][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-11-02 17:45:42 [2024-11-02T12:15:42,632][WARN ][logstash.licensechecker.licensereader] Health check failed {:code=>401, :url=>http://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
2024-11-02 17:45:42 [2024-11-02T12:15:42,647][WARN ][logstash.licensechecker.licensereader] Elasticsearch main endpoint returns 401 {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}"}
2024-11-02 17:45:42 [2024-11-02T12:15:42,648][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"Could not read Elasticsearch. Please check the credentials", :exception=>LogStash::ConfigurationError}
2024-11-02 17:45:42 [2024-11-02T12:15:42,697][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-11-02 17:45:42 [2024-11-02T12:15:42,712][WARN ][logstash.licensechecker.licensereader] Health check failed {:code=>401, :url=>http://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
2024-11-02 17:45:42 [2024-11-02T12:15:42,728][WARN ][logstash.licensechecker.licensereader] Elasticsearch main endpoint returns 401 {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}"}
2024-11-02 17:45:42 [2024-11-02T12:15:42,730][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Could not read Elasticsearch. Please check the credentials"}
2024-11-02 17:45:42 [2024-11-02T12:15:42,878][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:47 [2024-11-02T12:15:47,922][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:52 [2024-11-02T12:15:52,980][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:57 [2024-11-02T12:15:57,892][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:46:02 [2024-11-02T12:16:02,920][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:46:07 [2024-11-02T12:16:07,881][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:46:12 [2024-11-02T12:16:12,533][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-11-02 17:46:12 [2024-11-02T12:16:12,555][WARN ][logstash.licensechecker.licensereader] Health check failed {:code=>401, :url=>http://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
2024-11-02 17:46:12 [2024-11-02T12:16:12,571][WARN ][logstash.licensechecker.licensereader] Elasticsearch main endpoint returns 401 {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}"}
2024-11-02 17:46:12 [2024-11-02T12:16:12,578][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"Could not read Elasticsearch. Please check the credentials", :exception=>LogStash::ConfigurationError}
2024-11-02 17:46:12 [2024-11-02T12:16:12,619][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-11-02 17:46:12 [2024-11-02T12:16:12,634][WARN ][logstash.licensechecker.licensereader] Health check failed {:code=>401, :url=>http://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
2024-11-02 17:46:12 [2024-11-02T12:16:12,651][WARN ][logstash.licensechecker.licensereader] Elasticsearch main endpoint returns 401 {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}"}
2024-11-02 17:46:12 [2024-11-02T12:16:12,655][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Could not read Elasticsearch. Please check the credentials"}

here is my docker-compose.yml-:

version: '3'
services:
  # Frontend Angular Application
  frontend:
    image: pranchalm/easylogs_images:easylogs-frontend-1.0
    container_name: frontend-container
    ports:
      - "4200:80"
    networks:
      - app-network

  # Backend Spring Boot Application
  backend:
    image: pranchalm/easylogs_images:easylogs-backend-1.0
    container_name: backend-container
    depends_on:
      - elasticsearch
      - logstash
    ports:
      - "9090:9090"
    networks:
      - app-network

  # Elasticsearch
  elasticsearch:
    image: pranchalm/easylogs_images:mycustom-elasticsearch-1.0
    container_name: elasticsearch
    ports:
      - "9200:9200"
    networks:
      - app-network

  logstash:
    image: pranchalm/easylogs_images:mycustom-logstash-1.0
    container_name: logstash-container
    depends_on:
      - elasticsearch
    ports:
      - "5044:5044"
      - "9600:9600"
    networks:
      - app-network
    volumes:
      - /c/Micro_Services_ELK/logs:/usr/share/logstash/microservice1_logs
      - /c/Micro_Services_ELK/micro-service2logs:/usr/share/logstash/microservice2_logs

networks:
  app-network:
    driver: bridge

also here is the dockerfile used to built image of logstash-8.15.0-:

# Use the official Logstash base image 8.15.0
FROM docker.elastic.co/logstash/logstash:8.15.0

# Set environment variables for X-Pack management
ENV xpack.management.enabled=true
ENV xpack.management.pipeline.id="*"
ENV xpack.management.elasticsearch.username=elastic
ENV xpack.management.elasticsearch.password=elastic123
ENV xpack.management.elasticsearch.hosts="http://elasticsearch:9200"
ENV xpack.management.logstash.poll_interval=5s

# Expose necessary ports
EXPOSE 5044 9600

xpaxck.management.enabled =true as i am trying to use the logstash pipeline management feature.

credentials are supplied as we can see in Dockerfile while describing xpack.management part

when i tried by independently starting and checking by going inside the bash of the logstash container using curl -u elastic:elastic123 http://elasticsearch:9200, then its able to retrieve data but not when docker compose runs, then it exits.

security is enabled on elasticsearch side with basic-auth, license also enabled trial mode, able to retrive that data via postman too.

Please help me out on this, where is my configuration going wrong? Need support, thanks in advance.

2

Hello,

You do not have any configuration running in logstash:

2024-11-02 17:45:12 [2024-11-02T12:15:12,884][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:17 [2024-11-02T12:15:17,905][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:22 [2024-11-02T12:15:22,885][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:27 [2024-11-02T12:15:27,923][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:32 [2024-11-02T12:15:32,881][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-11-02 17:45:37 [2024-11-02T12:15:37,915][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.

Also, you are using a custom image, it is not clear what changes you made, you need to provide more context, but a 401 error means that the user or password is wrong, you need to double check it.

3

@leandrojmp thanks for the response, i have already provided the dockerfile in the previous post for logstash which is building the logstash image,

Regarding not having any configuration-:

Whatever configurations i require logstash to have i am supplying in the environment variables as shared previously, is there any issue with my docker-compose yml , or the dockerfile for logstash, i double checked the password and username it is fine, checked by curl and postman call, i am able to get a response from elasticsearch.

My concern is why it is not able to authorize, it should be able to.

If any more info needed from my side, so that it can help in issue diagnosis, please let me know. thaks

4

I don't think this is correct, you are not setting the correct names for the environment variables, check the documentation.

If I'm not wrong you need to set it as XPACK_MANAGEMENT_ENABLED=true for example, or logstash will not pick it up.

You would need to change all variables to this format.

5

Hi @leandrojmp ,corrected and tried with this dockerfile as suggested still issue is there, some logs have changed here are the latest run of logs, keeping docker-compse.yml as same as posted in previous message-:

Latest logs-:

2024-11-02 21:22:11 2024/11/02 15:52:11 Setting 'xpack.management.elasticsearch.username' from environment.
2024-11-02 21:22:11 2024/11/02 15:52:11 Setting 'xpack.management.elasticsearch.password' from environment.
2024-11-02 21:22:11 2024/11/02 15:52:11 Setting 'xpack.management.enabled' from environment.
2024-11-02 21:22:11 2024/11/02 15:52:11 Setting 'xpack.management.pipeline.id' from environment.
2024-11-02 21:22:11 2024/11/02 15:52:11 Setting 'xpack.management.elasticsearch.hosts' from environment.
2024-11-02 21:22:11 Using bundled JDK: /usr/share/logstash/jdk
2024-11-02 21:23:17 Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
2024-11-02 21:23:18 [2024-11-02T15:53:18,126][WARN ][deprecation.logstash.settings] The setting `http.host` is a deprecated alias for `api.http.host` and will be removed in a future release of Logstash. Please use api.http.host instead
2024-11-02 21:23:18 [2024-11-02T15:53:18,180][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
2024-11-02 21:23:18 [2024-11-02T15:53:18,183][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.15.0", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
2024-11-02 21:23:18 [2024-11-02T15:53:18,188][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
2024-11-02 21:23:18 [2024-11-02T15:53:18,203][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
2024-11-02 21:23:18 [2024-11-02T15:53:18,204][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
2024-11-02 21:23:18 [2024-11-02T15:53:18,241][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
2024-11-02 21:23:18 [2024-11-02T15:53:18,252][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
2024-11-02 21:23:18 [2024-11-02T15:53:18,302][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["*"], :poll_interval=>"TimeValue{duration=5, timeUnit=SECONDS}ns"}
2024-11-02 21:23:19 [2024-11-02T15:53:19,297][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}}
2024-11-02 21:23:19 [2024-11-02T15:53:19,601][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused>}
2024-11-02 21:23:19 [2024-11-02T15:53:19,609][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused"}
2024-11-02 21:23:19 [2024-11-02T15:53:19,646][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused>}
2024-11-02 21:23:19 [2024-11-02T15:53:19,650][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused {:url=>http://elastic:xxxxxx@elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
2024-11-02 21:23:19 [2024-11-02T15:53:19,656][WARN ][logstash.licensechecker.licensereader] Attempt to fetch Elasticsearch cluster info failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused"}
2024-11-02 21:23:19 [2024-11-02T15:53:19,686][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
2024-11-02 21:23:19 [2024-11-02T15:53:19,690][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
2024-11-02 21:23:19 [2024-11-02T15:53:19,736][ERROR][logstash.configmanagement.elasticsearchsource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
2024-11-02 21:23:19 [2024-11-02T15:53:19,746][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.>, :backtrace=>["/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:68:in `with_license_check'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:42:in `initialize'", "org/jruby/RubyClass.java:922:in `new'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:40:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:94:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:375:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:293:in `run'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:89:in `<main>'"]}
2024-11-02 21:23:19 [2024-11-02T15:53:19,767][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
2024-11-02 21:23:19 org.jruby.exceptions.SystemExit: (SystemExit) exit
2024-11-02 21:23:19     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
2024-11-02 21:23:19     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
2024-11-02 21:23:19     at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

Here is the achanged dockerfile for logstash as suggested

# Use the official Logstash base image 8.15.0
FROM docker.elastic.co/logstash/logstash:8.15.0

# Set environment variables for X-Pack management

ENV XPACK_MANAGEMENT_ENABLED=true
ENV XPACK_MANAGEMENT_PIPELINE_ID="*"
ENV XPACK_MANAGEMENT_ELASTICSEARCH_HOSTS="http://elasticsearch:9200"
ENV XPACK_MANAGEMENT_ELASTICSEARCH_USERNAME="elastic"
ENV XPACK_MANAGEMENT_ELASTICSEARCH_PASSWORD="elastic123"

# Expose necessary ports
EXPOSE 5044 9600

Context for diagnosis-: Steps done by me are as follows-:

  1. Running a docker compose yml with 4 containers as can be seen docker-compose.yml shared earlier

  2. Elasticsearch container is able to start and is up and running well, only logstash container exits in the whole compose

  3. My Expectancy-: Logstash docker container is able to connect to elasticSearch docker container and should be able to retrieve configured logstash pipelines and run them.

  4. In the current logs it is trying to retrieve license data by connecting to elasticsearch but it fails-: endpoitn formed to connect is also correct as it is "elasticsearch" which is configured in docker-compose.yml.

Please let me know if any more info from my side will be helpful, thanks for the support provided.

6

Your logstash container cannot connect to your elasticsearch.

Check the error logs:

2024-11-02 21:23:19 [2024-11-02T15:53:19,601][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.18.0.3] failed: Connection refused>}

Is your Elasticsearch running on http or https? You didn't share any information about your Elasticsearch configuration.

7

Hi @leandrojmp , here is the docker file used for constructing elasticsearch, http is used, security is enabled there using baisc-auth, no ssl, here is the docker file-:

# Use the official Elasticsearch 8.15 base image
FROM docker.elastic.co/elasticsearch/elasticsearch:8.15.0

# Set environment variables (you can customize these)
# Ensure the directory for data persistence is set correctly
ENV discovery.type=single-node
ENV ELASTIC_PASSWORD=elastic123
ENV xpack.security.enabled=true
ENV xpack.security.http.ssl.enabled=false
ENV xpack.security.transport.ssl.enabled=false
ENV xpack.license.self_generated.type=trial
ENV ES_PORT=9200

# Create a mount point for the data volume to persist indices
VOLUME ["/usr/share/elasticsearch/data"]

# Expose Elasticsearch port
EXPOSE 9200

this docker container is up and running and responding as expected.

8

The error in logstash is pretty clear, it cannot connect to http://elasticsearch:9200, there is not much to add about it.

You need to check that your elasticsearch is really running.

Please share some logs from the Elasticsearch container and some evidence that it is really running.

Like run curl http://your-docker-host:9200 -u elastic:elastic123 and share the result.

There is not much to troubleshoot here.

9

Hi @leandrojmp , as requested here are the logs from elasticsearch container up and running-:

2024-11-02 23:35:03 {"@timestamp":"2024-11-02T18:05:03.180Z", "log.level": "INFO", "message":"license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[e761abebc4f3][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"F0J9FuEPR3SKaQEBP8M42g","elasticsearch.node.id":"Vajt3ksbRiK8WI1WJaf2fg","elasticsearch.node.name":"e761abebc4f3","elasticsearch.cluster.name":"docker-cluster"}
2024-11-02 23:35:03 {"@timestamp":"2024-11-02T18:05:03.185Z", "log.level": "INFO", "message":"license [13683912-7e68-4576-a4f2-e7e242c17bba] mode [trial] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[e761abebc4f3][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"F0J9FuEPR3SKaQEBP8M42g","elasticsearch.node.id":"Vajt3ksbRiK8WI1WJaf2fg","elasticsearch.node.name":"e761abebc4f3","elasticsearch.cluster.name":"docker-cluster"}
2024-11-02 23:35:03 {"@timestamp":"2024-11-02T18:05:03.279Z", "log.level": "INFO", "message":"adding ingest pipeline metrics-apm.internal@default-pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[e761abebc4f3][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"F0J9FuEPR3SKaQEBP8M42g","elasticsearch.node.id":"Vajt3ksbRiK8WI1WJaf2fg","elasticsearch.node.name":"e761abebc4f3","elasticsearch.cluster.name":"docker-cluster"}
2024-11-02 23:35:03 {"@timestamp":"2024-11-02T18:05:03.279Z", "log.level": "INFO", "message":"adding ingest pipeline traces-apm@pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[e761abebc4f3][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"F0J9FuEPR3SKaQEBP8M42g","elasticsearch.node.id":"Vajt3ksbRiK8WI1WJaf2fg","elasticsearch.node.name":"e761abebc4f3","elasticsearch.cluster.name":"docker-cluster"}
2024-11-02 23:35:03 {"@timestamp":"2024-11-02T18:05:03.280Z", "log.level": "INFO", "message":"adding ingest pipeline metrics-apm@pipeline", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[e761abebc4f3][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.core.template.IndexTemplateRegistry","elasticsearch.cluster.uuid":"F0J9FuEPR3SKaQEBP8M42g","elasticsearch.node.id":"Vajt3ksbRiK8WI1WJaf2fg","elasticsearch.node.name":"e761abebc4f3","elasticsearch.cluster.name":"docker-cluster"}

also as requested, response from curl -u elastic:elastic123-:

curl http://localhost:9200 -u elastic:elastic123

{
  "name" : "e761abebc4f3",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "F0J9FuEPR3SKaQEBP8M42g",
  "version" : {
    "number" : "8.15.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "1a77947f34deddb41af25e6f0ddb8e830159c179",
    "build_date" : "2024-08-05T10:05:34.233336849Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

localhost-: As this command i ran on cmd, and docker has exposed elasticsearch to my machine at localhost:9200

But when logStash container is initiated as part of the docker-compose, the logstash container internally recognises elasticsearch container as -: "elasticsearch"- same as i have given in the servicename in docker-compose.yml

Response of curl is the evidence of elasticsearch up and running exposed to my machine on localhost:9200, but when logstash tries to connect to it, it will try to do that via servicename in the docker-compose.yml which is "elasticsearch".

I will double check configurations again from my side, i hope this info helps in diagnosing where things are going wrong .

10

Yeah, there is not much to troubleshoot here, it is a connection error on your docker setup.

This is really not a elasticsearch or logstash issue, it seems as a docker issue.

The error you have in Logstash is a connection error, not sure why because the containers are on the same network, but this is more related to docker than to Logstash.

I would recheck all the configurations and the logs.

11

@leandrojmp I really appreciate your help, but i would request to please share any findings on the docker side of things too, i understand that it may not be logstash or elasticsearch issue directly but how they behave in a docker-setup that is the area where things are right now.

If anyone else on the community side can help out on this, would be really helpful, kind of stuck on this for very long time.

12

Not sure if I understand the question, but they behave normally, if Logstash can communicate with Elasticsearch it will work, the issue here is that for some reason your Logstash container cannot connect to your Elasticsearch container.

This needs to be solved before you can use them, but this is a connectivity issue that needs to be solved first.

For example, did you try to connect to elasticsearch from the logstash container? What was the result?

From what you shared the compose seems ok, both containers are on the same network, so I would expect that they are able to talk with each other, but maybe something is not working right in your system.

I would suggest that you start a new compose, first add Elasticsearch, after it is running and up, add Logstash and see how the things go.