Dockerizing a working ELK project, 'Dead ES instance'

Elastic Stack Logstash
1

My setup works without Dockerizing. Now, making a Dockerized project, logstash cannot connect to the ES instance. I set http.host: "0.0.0.0" in logstash.yml, and my pipeline.conf is:

input {
  exec {
    command => "locale charmap"
    interval => 3600
  }
  tcp {
    host => "0.0.0.0"
    port => "5043"
    codec => plain {
        charset => "ISO-8859-1"
        # "UTF-8"
    }
}

}

filter {
  grok {
    match => { "message" => "(?<message>\[(.*?)\])" }
  }
  if "_grokparsefailure" in [tags] {
    drop { }
  }
}

output {
  elasticsearch {
    hosts => [ 'elasticsearch' ]
    index => "logstash-tcp-call-%{+YYYY.MM.dd}"
  }
  stdout{
    codec => rubydebug
  }
}

Maybe my host / dynamic routing is the issue, but I have tried the following possibilities:
[ 'elasticsearch' ]
[ 'elasticsearch:9200' ]
[ '192.168.1.7:9200' ]
[ '0.0.0.0:9200' ]

Would really appreciate helping my Logstash connect to ES! For some reason it says:

logstash         | [WARN ] 2020-08-04 01:59:58.622 [LogStash::Runner] licensereader - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://0.0.0.0:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://0.0.0.0:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
2

What does your Elasticsearch configuration look like? What do the logs of ES say? (The bootstrap checks might have failed.)
And just to understand your infrastructure: Logstash and ES are on two different servers in the same network and one is called elasticsearch?

3

Thank you for your prompt reply!
What does your Elasticsearch configuration look like?
So I have no explicit elasticsearch.yml, I have the following two yml:
logstash.yml:

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: "http://elasticsearch:9200"

pipelines.yml:

- pipeline.id: main
  path.config: "/usr/share/logstash/pipeline"

What do the logs of ES say? (The bootstrap checks might have failed.)
I will followup a reply with the log. (didn't see any failed bootstrap tests)

And just to understand your infrastructure: Logstash and ES are on two different servers in the same network and one is called elasticsearch?
That sounds accurate; to confirm, here is my docker-compose.yml:

version: "2.4"
services:

  elasticsearch:
      image: elasticsearch:7.8.0
      container_name: elasticsearch
      restart: always
      mem_reservation: 4000m
      mem_limit: 6000m
      environment:
        - discovery.type=single-node
        - MAX_MAP_COUNT=262144
      volumes:
        - ./elasticsearch/data:/usr/share/elasticsearch/data
      ports:
        - "9200:9200"
        - "9300:9300"
  
  logstash:
    image: docker.elastic.co/logstash/logstash:7.8.0
    container_name: logstash
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    restart: always
    mem_reservation: 2048m
    mem_limit: 4096m
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline/
      - ./logstash/config/:/usr/share/logstash/config/
    ports:
      - "9600:9600"
  
  kibana:
      image: docker.elastic.co/kibana/kibana:7.8.0
      container_name: kibana
      environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      restart: always
      ports:
        - "5601:5601"
4 

logstash         | [INFO ] 2020-08-04 15:17:48.633 [monitoring-license-manager] internalpipelinesource - Monitoring License OK
logstash         | [INFO ] 2020-08-04 15:17:48.633 [monitoring-license-manager] internalpipelinesource - Validated license for monitoring. Enabling monitoring pipeline.
logstash         | [INFO ] 2020-08-04 15:17:49.656 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash         | [WARN ] 2020-08-04 15:17:49.673 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
logstash         | [INFO ] 2020-08-04 15:17:49.688 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - ES Output version determined {:es_version=>7}
logstash         | [WARN ] 2020-08-04 15:17:49.688 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
logstash         | [INFO ] 2020-08-04 15:17:49.713 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elasticsearch:9200"]}
logstash         | [WARN ] 2020-08-04 15:17:49.714 [[.monitoring-logstash]-pipeline-manager] javapipeline - 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
logstash         | [INFO ] 2020-08-04 15:17:49.718 [[.monitoring-logstash]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x5837eeab@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:121 run>"}
logstash         | [INFO ] 2020-08-04 15:17:49.793 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>".monitoring-logstash"}
logstash         | [INFO ] 2020-08-04 15:17:49.802 [monitoring-license-manager] agent - Pipelines running {:count=>2, :running_pipelines=>[:main, :".monitoring-logstash"], :non_running_pipelines=>[]}
elasticsearch    | {"type": "server", "timestamp": "2020-08-04T15:19:18,117Z", "level": "INFO", "component": "o.e.m.j.JvmGcMonitorService", "cluster.name": "docker-cluster", "node.name": "09adc4351aef", "message": "[gc][145] overhead, spent [406ms] collecting in the last [1.5s]", "cluster.uuid": "ZvE8AxggQEmfWocw79CgCQ", "node.id": "2D0jukeSSM28Y_0wCpOyCg"  }
5 
logstash         | [INFO ] 2020-08-04 15:16:48.017 [LogStash::Runner] licensereader - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash         | [WARN ] 2020-08-04 15:16:48.528 [LogStash::Runner] licensereader - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash         | [WARN ] 2020-08-04 15:16:48.541 [LogStash::Runner] licensereader - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused) {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash         | [ERROR] 2020-08-04 15:16:48.543 [LogStash::Runner] licensereader - Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

Here is where logstash's connection to a dead ES instance failed

6

Thanks for sharing useful information with us.. It really helpful to me..I always prefer to read the quality content and this thing I found in you post. thanks for sharing with us..

7

Hello once again! I am still stuck on this issue unfortunately. Any tips on why the Dockerized version cannot connect Logstash to ES, but the non-docker version is fine? Thanks!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.