Unassigned shards docker swarm

Elastic Stack Elasticsearch
1

Hi guys,

I'm not an ELK expert.
I've a 2 nodes docker Swarm cluster in which I want to deploy the ELK stack.

This is my docker-compose.yml:

version: '3.4'

services:

  elk:
    image: docker.elastic.co/elasticsearch/elasticsearch-basic:6.2.1
    volumes:
      - ./elk/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
      - ./elk/data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xms256m -Xmx256m"
      ELASTIC_PASSWORD: changeme
    networks:
      - net
    deploy:
      mode: replicated
      replicas: 1
    
  logstash:
    image: docker.elastic.co/logstash/logstash:6.2.1
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "5000:5000"
      - "51415:51415"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - net
    deploy:
      mode: replicated
      replicas: 1

  kibana:
    image: docker.elastic.co/kibana/kibana:6.2.1
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
    ports:
      - "5601:5601"
    networks:
      - net
    deploy:
      mode: replicated
      replicas: 1

  logspout:
      image: gliderlabs/logspout:v3.2.4
      volumes:
        - '/var/run/docker.sock:/tmp/docker.sock'
      deploy:
        mode: global
      environment:
        SYSLOG_FORMAT: "rfc3164"
      command: 'syslog://logstash:51415'
      networks:
        - net

  apm-server:
      image: docker.elastic.co/apm/apm-server:6.2.0
      ports:
        - "8200:8200" 
      volumes:
        - ./apmserver/apm-server.yml:/usr/share/apm-server/apm-server.yml
      networks:
        - net
      deploy:
        mode: replicated
        replicas: 1

networks:
  net:

Basically I would like to forward all the docker containers logs to logstash. I'm doing it using logspout. Since in the docker swarm there is only the ELK stack running, the logs that logspout is forwarding to logstash are only the logs of the ELK stack containers.

It works fine for some hours, after that there is an exception: org.elasticsearch.action.UnavailableShardsException primary shard is not active Timeout

Output of GET _cat/shards?h=index,shard,prirep,state,unassigned.reason:

.kibana                           0 p STARTED    
.triggered_watches                0 p STARTED    
.monitoring-logstash-6-2018.02.16 0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-kibana-6-2018.02.19   0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-es-6-2018.02.18       0 p UNASSIGNED ALLOCATION_FAILED
.watches                          0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-logstash-6-2018.02.20 0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-logstash-6-2018.02.17 0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-es-6-2018.02.17       0 p UNASSIGNED ALLOCATION_FAILED
.watcher-history-7-2018.02.16     0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-kibana-6-2018.02.20   0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-es-6-2018.02.16       0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-logstash-6-2018.02.19 0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-es-6-2018.02.19       0 p UNASSIGNED ALLOCATION_FAILED
logstash-2018.02.16               0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-kibana-6-2018.02.16   0 p STARTED    
.monitoring-logstash-6-2018.02.18 0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-alerts-6              0 p STARTED    
.monitoring-kibana-6-2018.02.18   0 p UNASSIGNED ALLOCATION_FAILED
apm-6.2.0-2018.02.16              0 p STARTED    
.monitoring-kibana-6-2018.02.17   0 p UNASSIGNED ALLOCATION_FAILED
.monitoring-es-6-2018.02.20       0 p UNASSIGNED ALLOCATION_FAILED

Output of GET _template/logstash?pretty

{
  "logstash": {
    "order": 0,
    "index_patterns": [
      "logstash-*"
    ],
    "settings": {
      "index": {
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    },
   .....
  ......

Output of GET _cluster/health

{
  "cluster_name": "test-cluster",
  "status": "red",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 5,
  "active_shards": 5,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 17,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 22.727272727272727
}

Elasticsearch.yml

---
## Default Elasticsearch configuration from elasticsearch-docker.
## from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

What can I do to solve the problem?
thank you

2

No one that can help me?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.