My kibana has nothing but i think is elasticsearch problem

Elastic Stack Elasticsearch
1 
[2018-07-19T14:28:53,165][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [cvhieo8] failed to put mappings on indices [[[logstash-zww-ios-(2018-07-11)/n5LGnXYRQs2OaGVAuo9K8w]]], type [zww-ios]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
	at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$null$0(MasterService.java:122) ~[elasticsearch-6.1.2.jar:6.1.2]
	at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_162]
	at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:121) ~[elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2.jar:6.1.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
[2018-07-19T14:28:57,056][WARN ][o.e.m.j.JvmGcMonitorService] [cvhieo8] [gc][345] overhead, spent [3.4s] collecting in the last [3.8s]
[2018-07-19T14:28:57,062][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [cvhieo8] failed to put mappings on indices [[[logstash-zww-ios-(2018-07-11)/n5LGnXYRQs2OaGVAuo9K8w]]], type [zww-ios]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
	at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$null$0(MasterService.java:122) ~[elasticsearch-6.1.2.jar:6.1.2]
	at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_162]
	at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:121) ~[elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2.jar:6.1.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
[2018-07-19T14:29:00,545][WARN ][o.e.m.j.JvmGcMonitorService] [cvhieo8] [gc][346] overhead, spent [2.9s] collecting in the last [3.4s]

that elasticsearch log

[2018-07-19T14:30:14,910][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-07-19T14:30:33,133][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://localhost:9200/, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-07-19T14:30:33,134][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>8}
[2018-07-19T14:30:33,538][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://localhost:9200/, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-07-19T14:30:33,538][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>8}
[2018-07-19T14:30:34,992][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}

that logstash log

and kibana

Status: Red
Heap Total
273.86 MB
Heap Used
254.84 MB

what should i do

2

Can you provide the output if the Elasticsearch cluster stats API?

3 
 curl -XGET localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1961,
  "active_shards" : 1961,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1961,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

like that？

4

Well, I asked for the _cluster/stats API, but this still provides important information. It looks like the reason behind the poor performance probably is that you have too many shards. Please read this blog post around shards and sharding practices.

5

i stop the logstash ，elasticsearch from red to yellow
and
{
"_nodes": {
"total": 1,
"successful": 1,
"failed": 0
},
"cluster_name": "elasticsearch",
"timestamp": 1531990997640,
"status": "yellow",
"indices": {
"count": 393,
"shards": {
"total": 1961,
"primaries": 1961,
"replication": 0,
"index": {
"shards": {
"min": 1,
"max": 5,
"avg": 4.989821882951654
},
"primaries": {
"min": 1,
"max": 5,
"avg": 4.989821882951654
},
"replication": {
"min": 0,
"max": 0,
"avg": 0
}
}
},
"docs": {
"count": 13662244,
"deleted": 5
},
"store": {
"size": "2.5gb",
"size_in_bytes": 2697378996
},
"fielddata": {
"memory_size": "0b",
"memory_size_in_bytes": 0,
"evictions": 0
},
"query_cache": {
"memory_size": "0b",
"memory_size_in_bytes": 0,
"total_count": 0,
"hit_count": 0,
"miss_count": 0,
"cache_size": 0,
"cache_count": 0,
"evictions": 0
},
"completion": {
"size": "0b",
"size_in_bytes": 0
},
"segments": {
"count": 8516,
"memory": "70.9mb",
"memory_in_bytes": 74411917,
"terms_memory": "59.5mb",
"terms_memory_in_bytes": 62494634,
"stored_fields_memory": "3.1mb",
"stored_fields_memory_in_bytes": 3306000,
"term_vectors_memory": "0b",
"term_vectors_memory_in_bytes": 0,
"norms_memory": "3.1mb",
"norms_memory_in_bytes": 3285760,
"points_memory": "265.2kb",
"points_memory_in_bytes": 271587,
"doc_values_memory": "4.8mb",
"doc_values_memory_in_bytes": 5053936,
"index_writer_memory": "0b",
"index_writer_memory_in_bytes": 0,
"version_map_memory": "0b",
"version_map_memory_in_bytes": 0,
"fixed_bit_set": "0b",
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": 1531981923919,
"file_sizes": {}
}
},
"nodes": {
"count": {
"total": 1,
"data": 1,
"coordinating_only": 0,
"master": 1,
"ingest": 1
},
"versions": [
"6.1.2"
],
"os": {
"available_processors": 4,
"allocated_processors": 4,
"names": [
{
"name": "Linux",
"count": 1
}
],
"mem": {
"total": "7.6gb",
"total_in_bytes": 8202559488,
"free": "592.9mb",
"free_in_bytes": 621764608,
"used": "7gb",
"used_in_bytes": 7580794880,
"free_percent": 8,
"used_percent": 92
}
},
"process": {
"cpu": {
"percent": 1
},
"open_file_descriptors": {
"min": 6149,
"max": 6149,
"avg": 6149
}
},
"jvm": {
"max_uptime": "12.9m",
"max_uptime_in_millis": 776422,
"versions": [
{
"version": "1.8.0_162",
"vm_name": "Java HotSpot(TM) 64-Bit Server VM",
"vm_version": "25.162-b12",
"vm_vendor": "Oracle Corporation",
"count": 1
}
],
"mem": {
"heap_used": "1gb",
"heap_used_in_bytes": 1128987096,
"heap_max": "1.9gb",
"heap_max_in_bytes": 2112618496
},
"threads": 52
},
"fs": {
"total": "492gb",
"total_in_bytes": 528310599680,
"free": "487.7gb",
"free_in_bytes": 523675340800,
"available": "462.6gb",
"available_in_bytes": 496815017984
},
"plugins": [],
"network_types": {
"transport_types": {
"netty4": 1
},
"http_types": {
"netty4": 1
}
}
}
}

6

_cluster/stars

{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "elasticsearch",
  "timestamp": 1531991357797,
  "status": "yellow",
  "indices": {
    "count": 393,
    "shards": {
      "total": 1961,
      "primaries": 1961,
      "replication": 0,
      "index": {
        "shards": {
          "min": 1,
          "max": 5,
          "avg": 4.989821882951654
        },
        "primaries": {
          "min": 1,
          "max": 5,
          "avg": 4.989821882951654
        },
        "replication": {
          "min": 0,
          "max": 0,
          "avg": 0
        }
      }
    },
    "docs": {
      "count": 13662244,
      "deleted": 5
    },
    "store": {
      "size_in_bytes": 2697378996
    },
    "fielddata": {
      "memory_size_in_bytes": 0,
      "evictions": 0
    },
    "query_cache": {
      "memory_size_in_bytes": 0,
      "total_count": 0,
      "hit_count": 0,
      "miss_count": 0,
      "cache_size": 0,
      "cache_count": 0,
      "evictions": 0
    },
    "completion": {
      "size_in_bytes": 0
    },
    "segments": {
      "count": 8516,
      "memory_in_bytes": 74411917,
      "terms_memory_in_bytes": 62494634,
      "stored_fields_memory_in_bytes": 3306000,
      "term_vectors_memory_in_bytes": 0,
      "norms_memory_in_bytes": 3285760,
      "points_memory_in_bytes": 271587,
      "doc_values_memory_in_bytes": 5053936,
      "index_writer_memory_in_bytes": 0,
      "version_map_memory_in_bytes": 0,
      "fixed_bit_set_memory_in_bytes": 0,
      "max_unsafe_auto_id_timestamp": 1531981923919,
      "file_sizes": {}
    }
  },
  "nodes": {
    "count": {
      "total": 1,
      "data": 1,
      "coordinating_only": 0,
      "master": 1,
      "ingest": 1
    },
    "versions": [
      "6.1.2"
    ],
    "os": {
      "available_processors": 4,
      "allocated_processors": 4,
      "names": [
        {
          "name": "Linux",
          "count": 1
        }
      ],
      "mem": {
        "total_in_bytes": 8202559488,
        "free_in_bytes": 547627008,
        "used_in_bytes": 7654932480,
        "free_percent": 7,
        "used_percent": 93
      }
    },
    "process": {
      "cpu": {
        "percent": 2
      },
      "open_file_descriptors": {
        "min": 6149,
        "max": 6149,
        "avg": 6149
      }
    },
    "jvm": {
      "max_uptime_in_millis": 1138367,
      "versions": [
        {
          "version": "1.8.0_162",
          "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
          "vm_version": "25.162-b12",
          "vm_vendor": "Oracle Corporation",
          "count": 1
        }
      ],
      "mem": {
        "heap_used_in_bytes": 1108124712,
        "heap_max_in_bytes": 2112618496
      },
      "threads": 53
    },
    "fs": {
      "total_in_bytes": 528310599680,
      "free_in_bytes": 523675340800,
      "available_in_bytes": 496815017984
    },
    "plugins": [],
    "network_types": {
      "transport_types": {
        "netty4": 1
      },
      "http_types": {
        "netty4": 1
      }
    }
  }
}
7

You have far too many shards for a single node cluster with just 2GB heap. You will need to reduce this dramatically.

8

how can i do that

9

You will probably need to change how you shard the data and reindex it. If you are using daily indices, instead go for weekly or monthly indices, possibly with a single primary shard, assuming your retention period can handle that granularity. Aim for an average shard size around 10GB or larger.

10

now i turn my heap to 3g and i change my index generation rule ，it generate quarterly.
and i close about 1200 shards,
heap.current heap.percent heap.max ram.max disk.avail node.role m
797.3mb 26 2.9gb 7.6gb 462.5gb mdi *
now my problem is i can't keep them close.i want ask some way to reduce index or shards.
my index generate by logstash ,if i reindex,i can't imagine the workload

11

and i can't find where to change the default shard number ,

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.