Logstash cannot connect to elasticsearch

Elastic Stack Logstash
1

I am not able to connect logstash with elasticsearch, below are my configuration files

version: "1.0"
services:
  elasticsearch:
    container_name: elasticsearch-container
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.3
    environment:
      - "discovery.type=single-node"
      - ELASTIC_PASSWORD=password
    networks:
      - elk-bridge-network
    ports:
      - 9200:9200
  logstash:
    container_name: logstash-container
    image: docker.elastic.co/logstash/logstash:8.15.3
    networks:
      - elk-bridge-network
    volumes:
      - ./config/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
    environment:
      - ELASTICSEARCH_USER=elastic
      - ELASTICSEARCH_PASSWORD=password
      - ELASTICSEARCH_URL=https://elasticsearch:9200
    depends_on:
      - elasticsearch
    ports:
      - 5044:5044
  kibana:
    container_name: kibana-container
    image: docker.elastic.co/kibana/kibana:8.15.3
    networks:
      - elk-bridge-network
    depends_on:
      - elasticsearch
    ports:
      - 5601:5601
networks:
  elk-bridge-network:
    driver: bridge

logstash.conf file

input {
  tcp {
    port => 5044
    codec => json_lines
  }
}

output {
  elasticsearch {
    hosts => ["${ELASTICSEARCH_URL}"]
    index => "logstash-%{+YYYY.MM.dd}"
    user => "${ELASTICSEARCH_USER}"
    password => "${ELASTICSEARCH_PASSWORD}"
    ssl_verification_mode => "none"
  }
}

logs im getting

2024-10-28 10:31:17 Please configure Elastic Agent to monitor Logstash. Documentation can be found at: 
2024-10-28 10:31:17 https://www.elastic.co/guide/en/logstash/current/monitoring-with-elastic-agent.html
2024-10-28 10:31:17 [2024-10-28T05:31:17,821][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-10-28 10:31:17 [2024-10-28T05:31:17,854][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/192.168.208.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/192.168.208.2] failed: Connection refused>}
2024-10-28 10:31:17 [2024-10-28T05:31:17,856][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.208.2] failed: Connection refused"}
2024-10-28 10:31:17 [2024-10-28T05:31:17,869][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/192.168.208.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/192.168.208.2] failed: Connection refused>}
2024-10-28 10:31:17 [2024-10-28T05:31:17,871][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.208.2] failed: Connection refused {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.208.2] failed: Connection refused", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
2024-10-28 10:31:17 [2024-10-28T05:31:17,874][WARN ][logstash.licensechecker.licensereader] Attempt to fetch Elasticsearch cluster info failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.208.2] failed: Connection refused"}
2024-10-28 10:31:17 [2024-10-28T05:31:17,897][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
2

I'm surprised to see http://elasticsearch:9200/ in the logs as you mentioned ELASTICSEARCH_URL=https://elasticsearch:9200 in the docker compose file.

Could you check that what you shared is exactly what you used?

3

Well I rechecked, the docker-compose.yml was not saved apparently. I saved and re-created the containers. still getting error

2024-10-28 17:43:47 Using bundled JDK: /usr/share/logstash/jdk
2024-10-28 17:44:01 Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
2024-10-28 17:44:01 [2024-10-28T12:44:01,573][WARN ][deprecation.logstash.settings] The setting `http.host` is a deprecated alias for `api.http.host` and will be removed in a future release of Logstash. Please use api.http.host instead
2024-10-28 17:44:01 [2024-10-28T12:44:01,604][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
2024-10-28 17:44:01 [2024-10-28T12:44:01,610][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.15.3", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [aarch64-linux]"}
2024-10-28 17:44:01 [2024-10-28T12:44:01,613][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
2024-10-28 17:44:01 [2024-10-28T12:44:01,620][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
2024-10-28 17:44:01 [2024-10-28T12:44:01,620][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
2024-10-28 17:44:01 [2024-10-28T12:44:01,634][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
2024-10-28 17:44:01 [2024-10-28T12:44:01,641][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
2024-10-28 17:44:01 [2024-10-28T12:44:01,928][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"88a629ab-5ed3-4fd6-8b15-1ab86ed3684a", :path=>"/usr/share/logstash/data/uuid"}
2024-10-28 17:44:03 [2024-10-28T12:44:03,177][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
2024-10-28 17:44:03 [2024-10-28T12:44:03,189][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
2024-10-28 17:44:03 Please configure Elastic Agent to monitor Logstash. Documentation can be found at: 
2024-10-28 17:44:03 https://www.elastic.co/guide/en/logstash/current/monitoring-with-elastic-agent.html
2024-10-28 17:44:04 [2024-10-28T12:44:04,772][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2024-10-28 17:44:04 [2024-10-28T12:44:04,986][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused>}
2024-10-28 17:44:04 [2024-10-28T12:44:04,995][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused"}
2024-10-28 17:44:05 [2024-10-28T12:44:05,016][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused>}
2024-10-28 17:44:05 [2024-10-28T12:44:05,016][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
2024-10-28 17:44:05 [2024-10-28T12:44:05,020][WARN ][logstash.licensechecker.licensereader] Attempt to fetch Elasticsearch cluster info failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused"}
2024-10-28 17:44:05 [2024-10-28T12:44:05,043][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
2024-10-28 17:44:05 [2024-10-28T12:44:05,055][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
2024-10-28 17:44:05 [2024-10-28T12:44:05,114][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
2024-10-28 17:44:05 [2024-10-28T12:44:05,396][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2024-10-28 17:44:06 [2024-10-28T12:44:06,781][INFO ][org.reflections.Reflections] Reflections took 524 ms to scan 1 urls, producing 138 keys and 481 values
2024-10-28 17:44:07 [2024-10-28T12:44:07,458][INFO ][logstash.codecs.jsonlines] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
2024-10-28 17:44:07 [2024-10-28T12:44:07,684][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
2024-10-28 17:44:07 [2024-10-28T12:44:07,721][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://elasticsearch:9200"]}
2024-10-28 17:44:07 [2024-10-28T12:44:07,726][WARN ][logstash.outputs.elasticsearch][main] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set `ssl_verification_mode => full`
2024-10-28 17:44:07 [2024-10-28T12:44:07,751][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@elasticsearch:9200/]}}
2024-10-28 17:44:07 [2024-10-28T12:44:07,838][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused>}
2024-10-28 17:44:07 [2024-10-28T12:44:07,839][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused"}
2024-10-28 17:44:07 [2024-10-28T12:44:07,863][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"logstash-%{+YYYY.MM.dd}"}
2024-10-28 17:44:07 [2024-10-28T12:44:07,871][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
2024-10-28 17:44:07 [2024-10-28T12:44:07,918][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1250, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x14c10b5d /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
2024-10-28 17:44:09 [2024-10-28T12:44:09,092][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.17}
2024-10-28 17:44:09 [2024-10-28T12:44:09,224][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
2024-10-28 17:44:09 [2024-10-28T12:44:09,227][INFO ][logstash.inputs.tcp      ][main][eea5192a76575cbf28bf59c9e8d08a887ec382c86d2f3d1d04988cee27d6fa25] Starting tcp input listener {:address=>"0.0.0.0:5044", :ssl_enabled=>false}
2024-10-28 17:44:09 [2024-10-28T12:44:09,236][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
2024-10-28 17:44:12 [2024-10-28T12:44:12,947][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused>}
2024-10-28 17:44:12 [2024-10-28T12:44:12,957][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused"}
2024-10-28 17:44:18 [2024-10-28T12:44:17,992][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused>}
2024-10-28 17:44:18 [2024-10-28T12:44:17,999][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused"}
2024-10-28 17:44:23 [2024-10-28T12:44:23,020][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused>}
2024-10-28 17:44:23 [2024-10-28T12:44:23,022][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/192.168.240.2] failed: Connection refused"}
2024-10-28 17:44:28 [2024-10-28T12:44:28,922][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@elasticsearch:9200/"}
2024-10-28 17:44:28 [2024-10-28T12:44:28,978][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.15.3) {:es_version=>8}
2024-10-28 17:44:28 [2024-10-28T12:44:28,978][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
4

These lines means that your logstash connected to your Elasticsearch.

Do you have any warn or error logs after that?

5

It was giving error for no valid certificate found. What I did, copy the http_ca.crt file from elasticsearch_container and placed in /certs/ca folder and mount to usr/share/logstash/config/certs. Now it is giving below error

2024-10-28 18:32:00 [2024-10-28T13:32:00,966][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.20.0.2] failed: Connection refused"}
2024-10-28 18:32:06 [2024-10-28T13:32:06,924][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@elasticsearch:9200/"}
2024-10-28 18:32:06 [2024-10-28T13:32:06,989][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.15.3) {:es_version=>8}
2024-10-28 18:32:06 [2024-10-28T13:32:06,990][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
2024-10-28 18:32:20 [2024-10-28T13:32:20,348][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
2024-10-28 18:32:20 [2024-10-28T13:32:20,353][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
2024-10-28 18:32:20 [2024-10-28T13:32:20,498][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch:9200 failed to respond", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::OrgApacheHttp::NoHttpResponseException: elasticsearch:9200 failed to respond>}
2024-10-28 18:32:20 [2024-10-28T13:32:20,502][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ClientProtocolException] elasticsearch:9200 failed to respond"}
2024-10-28 18:32:20 [2024-10-28T13:32:20,930][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
2024-10-28 18:32:21 [2024-10-28T13:32:21,040][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"}
2024-10-28 18:32:50 [2024-10-28T13:32:50,345][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
2024-10-28 18:32:50 [2024-10-28T13:32:50,346][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
2024-10-28 18:32:50 [2024-10-28T13:32:50,582][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch:9200 failed to respond", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::OrgApacheHttp::NoHttpResponseException: elasticsearch:9200 failed to respond>}
2024-10-28 18:32:50 [2024-10-28T13:32:50,584][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ClientProtocolException] elasticsearch:9200 failed to respond"}
2024-10-28 18:33:20 [2024-10-28T13:33:20,345][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
2024-10-28 18:33:20 [2024-10-28T13:33:20,347][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
2024-10-28 18:33:20 [2024-10-28T13:33:20,681][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch:9200 failed to respond", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::OrgApacheHttp::NoHttpResponseException: elasticsearch:9200 failed to respond>}
2024-10-28 18:33:20 [2024-10-28T13:33:20,683][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ClientProtocolException] elasticsearch:9200 failed to respond"}
2024-10-28 18:33:50 [2024-10-28T13:33:50,360][ERROR][lo`
6

Something is not correct in the configuration, it is still trying to connect using http but you are using https

Do you have anything else in logstash.conf ?

7

No I dont have anything yet, it is same as shared above

8

I believe it is issue with certificates as ssl is enabled by default in 8.x versions. If anyone can guide me how to add elasticsearch certificate for logstash. As previously mentioned I imported elasticsearch container http_ca.cert file and created a logstash container volume.
directory structure is
certs ->ca -> ca.cert

logstash:
    container_name: logstash-container
    image: docker.elastic.co/logstash/logstash:8.15.3
    networks:
      - elk-bridge-network
    volumes:
      - ./config/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
      - ./certs:/usr/share/logstash/config/certs
    environment:
      - ELASTICSEARCH_USER=elastic
      - ELASTICSEARCH_PASSWORD=password
      - ELASTICSEARCH_URL=https://elasticsearch:9200
    depends_on:
      - elasticsearch
    ports:
      - 5044:5044
9

I don't think this is related to certificate issues, the error is clearing saying that your logstash is trying to connect to elasticserch using http.

Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/]

This means that you have at least one output configured to use http when connecting to Elasticsearch.

You need to doublecheck that you are indeed using the configuration you shared.

I suggest that you connect directly to the logstash container and check the /usr/share/logstash/pipeline/logstash.conf file to confirm that it is using the configuration you shared.

10

Yes, there was default logstash.yml that was pointing towards http. I updated it and now again getting error about certificatates

2024-10-28 21:53:24 [2024-10-28T16:53:24,112][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.22.0.2] failed: Connection refused"}
2024-10-28 21:53:29 [2024-10-28T16:53:29,666][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@elasticsearch:9200/"}
2024-10-28 21:53:29 [2024-10-28T16:53:29,697][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.15.3) {:es_version=>8}
2024-10-28 21:53:29 [2024-10-28T16:53:29,697][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
2024-10-28 21:53:37 [2024-10-28T16:53:37,599][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
2024-10-28 21:53:37 [2024-10-28T16:53:37,604][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
2024-10-28 21:53:37 [2024-10-28T16:53:37,754][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
2024-10-28 21:53:37 [2024-10-28T16:53:37,755][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
2024-10-28 21:53:38 [2024-10-28T16:53:38,932][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
2024-10-28 21:53:38 [2024-10-28T16:53:38,992][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"}
2024-10-28 21:54:07 [2024-10-28T16:54:07,598][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
2024-10-28 21:54:07 [2024-10-28T16:54:07,601][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
2024-10-28 21:54:07 [2024-10-28T16:54:07,879][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
2024-10-28 21:54:07 [2024-10-28T16:54:07,880][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}