Docker-compose Lostash "Attempted to resurrect connection to dead ES instance, but got an error"

Elastic Stack Logstash
1

Wanting to build my Elastic stack from docker-compose.
Here my file.

version: '3.7'
services:
  elasticsearch1:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
    container_name: Elasticsearch1
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet

  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
    container_name: Elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch1"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

  kibana:
    image: docker.elastic.co/kibana/kibana:6.3.2
    container_name: Kibana
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://elasticsearch1:9200
    networks:
      - esnet
    depends_on:
      - elasticsearch1

  logstash:
    image: docker.elastic.co/logstash/logstash:6.3.2
    container_name: Logstash
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    ports:
      - "5959:5959"
    volumes:
      - ./logstash/:/etc/logstash/conf.d
    depends_on:
      - elasticsearch1
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local

networks:
  esnet:

Everything is working except for Logstash that cannot get to Elasticsearch.

Here my logstash.conf:

input { stdin { } }

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch { hosts => ["elasticsearch1", "elasticsearch2"] }
  stdout { codec => rubydebug }
}
2

I modified a bit my docker file and now Logstash exit w/o any error.

Logstash          | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
Logstash          | [2018-07-30T15:26:19,358][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
Logstash          | [2018-07-30T15:26:19,367][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
Logstash          | [2018-07-30T15:26:19,923][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"32988539-4e45-4bfe-ac92-0d0d9f753267", :path=>"/usr/share/logstash/data/uuid"}
Logstash          | [2018-07-30T15:26:20,454][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
Logstash          | [2018-07-30T15:26:23,116][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
Logstash          | [2018-07-30T15:26:23,602][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch1:9200/, http://elasticsearch2:9200/]}}
Logstash          | [2018-07-30T15:26:23,613][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch1:9200/, :path=>"/"}
Logstash          | [2018-07-30T15:26:23,842][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch1:9200/"}
Logstash          | [2018-07-30T15:26:23,927][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
Logstash          | [2018-07-30T15:26:23,931][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
Logstash          | [2018-07-30T15:26:23,933][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch2:9200/, :path=>"/"}
Logstash          | [2018-07-30T15:26:24,022][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch2:9200/"}
Logstash          | [2018-07-30T15:26:24,104][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch1", "//elasticsearch2"]}
Logstash          | [2018-07-30T15:26:24,123][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
Logstash          | [2018-07-30T15:26:24,145][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
Logstash          | [2018-07-30T15:26:24,276][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x364dd829 run>"}
Logstash          | [2018-07-30T15:26:24,377][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Logstash          | [2018-07-30T15:26:24,680][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
Logstash          | [2018-07-30T15:26:25,106][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x364dd829 run>"}
Logstash exited with code 0
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.