# My kibana has nothing but i think is elasticsearch problem
**URL:** https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661
**Category:** Elasticsearch
**Created:** [July 19, 2018, 6:48am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661 "2018-07-19T06:48:01Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@chenzilong444](https://discuss.elastic.co/u/chenzilong444)
#### Post date: [July 19, 2018, 6:48am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/1 "2018-07-19T06:48:01Z")
```
[2018-07-19T14:28:53,165][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [cvhieo8] failed to put mappings on indices [[[logstash-zww-ios-(2018-07-11)/n5LGnXYRQs2OaGVAuo9K8w]]], type [zww-ios]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$null$0(MasterService.java:122) ~[elasticsearch-6.1.2.jar:6.1.2]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_162]
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:121) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2.jar:6.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
[2018-07-19T14:28:57,056][WARN][o.e.m.j.JvmGcMonitorService] [cvhieo8] [gc][345] overhead, spent [3.4s] collecting in the last [3.8s]
[2018-07-19T14:28:57,062][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [cvhieo8] failed to put mappings on indices [[[logstash-zww-ios-(2018-07-11)/n5LGnXYRQs2OaGVAuo9K8w]]], type [zww-ios]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$null$0(MasterService.java:122) ~[elasticsearch-6.1.2.jar:6.1.2]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_162]
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:121) ~[elasticsearch-6.1.2.jar:6.1.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2.jar:6.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
[2018-07-19T14:29:00,545][WARN][o.e.m.j.JvmGcMonitorService] [cvhieo8] [gc][346] overhead, spent [2.9s] collecting in the last [3.4s]
```
that elasticsearch log
```
[2018-07-19T14:30:14,910][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-07-19T14:30:33,133][WARN][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://localhost:9200/, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-07-19T14:30:33,134][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>8}
[2018-07-19T14:30:33,538][WARN][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://localhost:9200/, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-07-19T14:30:33,538][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>8}
[2018-07-19T14:30:34,992][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
```
that logstash log
and kibana
Status: Red
Heap Total
273.86 MB
Heap Used
254.84 MB
what should i do
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 19, 2018, 7:04am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/2 "2018-07-19T07:04:20Z")
Can you provide the output if the Elasticsearch [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/cluster-stats.html)?
---
### Author:  [@chenzilong444](https://discuss.elastic.co/u/chenzilong444)
#### Post date: [July 19, 2018, 7:21am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/3 "2018-07-19T07:21:06Z")
```
curl -XGET localhost:9200/_cluster/health?pretty=true
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1961,
"active_shards" : 1961,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1961,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}
```
like that？
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 19, 2018, 7:28am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/4 "2018-07-19T07:28:48Z")
Well, I asked for the `_cluster/stats` API, but this still provides important information. It looks like the reason behind the poor performance probably is that you have too many shards. Please read [this blog post around shards and sharding practices](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster).
---
### Author:  [@chenzilong444](https://discuss.elastic.co/u/chenzilong444)
#### Post date: [July 19, 2018, 9:04am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/5 "2018-07-19T09:04:53Z")
i stop the logstash ，elasticsearch from red to yellow
and
{
"\_nodes": {
"total": 1,
"successful": 1,
"failed": 0
},
"cluster\_name": "elasticsearch",
"timestamp": 1531990997640,
"status": "yellow",
"indices": {
"count": 393,
"shards": {
"total": 1961,
"primaries": 1961,
"replication": 0,
"index": {
"shards": {
"min": 1,
"max": 5,
"avg": 4.989821882951654
},
"primaries": {
"min": 1,
"max": 5,
"avg": 4.989821882951654
},
"replication": {
"min": 0,
"max": 0,
"avg": 0
}
}
},
"docs": {
"count": 13662244,
"deleted": 5
},
"store": {
"size": "2.5gb",
"size\_in\_bytes": 2697378996
},
"fielddata": {
"memory\_size": "0b",
"memory\_size\_in\_bytes": 0,
"evictions": 0
},
"query\_cache": {
"memory\_size": "0b",
"memory\_size\_in\_bytes": 0,
"total\_count": 0,
"hit\_count": 0,
"miss\_count": 0,
"cache\_size": 0,
"cache\_count": 0,
"evictions": 0
},
"completion": {
"size": "0b",
"size\_in\_bytes": 0
},
"segments": {
"count": 8516,
"memory": "70.9mb",
"memory\_in\_bytes": 74411917,
"terms\_memory": "59.5mb",
"terms\_memory\_in\_bytes": 62494634,
"stored\_fields\_memory": "3.1mb",
"stored\_fields\_memory\_in\_bytes": 3306000,
"term\_vectors\_memory": "0b",
"term\_vectors\_memory\_in\_bytes": 0,
"norms\_memory": "3.1mb",
"norms\_memory\_in\_bytes": 3285760,
"points\_memory": "265.2kb",
"points\_memory\_in\_bytes": 271587,
"doc\_values\_memory": "4.8mb",
"doc\_values\_memory\_in\_bytes": 5053936,
"index\_writer\_memory": "0b",
"index\_writer\_memory\_in\_bytes": 0,
"version\_map\_memory": "0b",
"version\_map\_memory\_in\_bytes": 0,
"fixed\_bit\_set": "0b",
"fixed\_bit\_set\_memory\_in\_bytes": 0,
"max\_unsafe\_auto\_id\_timestamp": 1531981923919,
"file\_sizes": {}
}
},
"nodes": {
"count": {
"total": 1,
"data": 1,
"coordinating\_only": 0,
"master": 1,
"ingest": 1
},
"versions": [
"6.1.2"
],
"os": {
"available\_processors": 4,
"allocated\_processors": 4,
"names": [
{
"name": "Linux",
"count": 1
}
],
"mem": {
"total": "7.6gb",
"total\_in\_bytes": 8202559488,
"free": "592.9mb",
"free\_in\_bytes": 621764608,
"used": "7gb",
"used\_in\_bytes": 7580794880,
"free\_percent": 8,
"used\_percent": 92
}
},
"process": {
"cpu": {
"percent": 1
},
"open\_file\_descriptors": {
"min": 6149,
"max": 6149,
"avg": 6149
}
},
"jvm": {
"max\_uptime": "12.9m",
"max\_uptime\_in\_millis": 776422,
"versions": [
{
"version": "1.8.0\_162",
"vm\_name": "Java HotSpot(TM) 64-Bit Server VM",
"vm\_version": "25.162-b12",
"vm\_vendor": "Oracle Corporation",
"count": 1
}
],
"mem": {
"heap\_used": "1gb",
"heap\_used\_in\_bytes": 1128987096,
"heap\_max": "1.9gb",
"heap\_max\_in\_bytes": 2112618496
},
"threads": 52
},
"fs": {
"total": "492gb",
"total\_in\_bytes": 528310599680,
"free": "487.7gb",
"free\_in\_bytes": 523675340800,
"available": "462.6gb",
"available\_in\_bytes": 496815017984
},
"plugins": [],
"network\_types": {
"transport\_types": {
"netty4": 1
},
"http\_types": {
"netty4": 1
}
}
}
}
---
### Author:  [@chenzilong444](https://discuss.elastic.co/u/chenzilong444)
#### Post date: [July 19, 2018, 9:10am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/6 "2018-07-19T09:10:11Z")
\_cluster/stars
```
{
"_nodes": {
"total": 1,
"successful": 1,
"failed": 0
},
"cluster_name": "elasticsearch",
"timestamp": 1531991357797,
"status": "yellow",
"indices": {
"count": 393,
"shards": {
"total": 1961,
"primaries": 1961,
"replication": 0,
"index": {
"shards": {
"min": 1,
"max": 5,
"avg": 4.989821882951654
},
"primaries": {
"min": 1,
"max": 5,
"avg": 4.989821882951654
},
"replication": {
"min": 0,
"max": 0,
"avg": 0
}
}
},
"docs": {
"count": 13662244,
"deleted": 5
},
"store": {
"size_in_bytes": 2697378996
},
"fielddata": {
"memory_size_in_bytes": 0,
"evictions": 0
},
"query_cache": {
"memory_size_in_bytes": 0,
"total_count": 0,
"hit_count": 0,
"miss_count": 0,
"cache_size": 0,
"cache_count": 0,
"evictions": 0
},
"completion": {
"size_in_bytes": 0
},
"segments": {
"count": 8516,
"memory_in_bytes": 74411917,
"terms_memory_in_bytes": 62494634,
"stored_fields_memory_in_bytes": 3306000,
"term_vectors_memory_in_bytes": 0,
"norms_memory_in_bytes": 3285760,
"points_memory_in_bytes": 271587,
"doc_values_memory_in_bytes": 5053936,
"index_writer_memory_in_bytes": 0,
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": 1531981923919,
"file_sizes": {}
}
},
"nodes": {
"count": {
"total": 1,
"data": 1,
"coordinating_only": 0,
"master": 1,
"ingest": 1
},
"versions": [
"6.1.2"
],
"os": {
"available_processors": 4,
"allocated_processors": 4,
"names": [
{
"name": "Linux",
"count": 1
}
],
"mem": {
"total_in_bytes": 8202559488,
"free_in_bytes": 547627008,
"used_in_bytes": 7654932480,
"free_percent": 7,
"used_percent": 93
}
},
"process": {
"cpu": {
"percent": 2
},
"open_file_descriptors": {
"min": 6149,
"max": 6149,
"avg": 6149
}
},
"jvm": {
"max_uptime_in_millis": 1138367,
"versions": [
{
"version": "1.8.0_162",
"vm_name": "Java HotSpot(TM) 64-Bit Server VM",
"vm_version": "25.162-b12",
"vm_vendor": "Oracle Corporation",
"count": 1
}
],
"mem": {
"heap_used_in_bytes": 1108124712,
"heap_max_in_bytes": 2112618496
},
"threads": 53
},
"fs": {
"total_in_bytes": 528310599680,
"free_in_bytes": 523675340800,
"available_in_bytes": 496815017984
},
"plugins": [],
"network_types": {
"transport_types": {
"netty4": 1
},
"http_types": {
"netty4": 1
}
}
}
}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 19, 2018, 9:13am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/7 "2018-07-19T09:13:34Z")
You have far too many shards for a single node cluster with just 2GB heap. You will need to reduce this dramatically.
---
### Author:  [@chenzilong444](https://discuss.elastic.co/u/chenzilong444)
#### Post date: [July 19, 2018, 9:20am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/8 "2018-07-19T09:20:20Z")
how can i do that
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 19, 2018, 9:22am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/9 "2018-07-19T09:22:56Z")
You will probably need to change how you shard the data and reindex it. If you are using daily indices, instead go for weekly or monthly indices, possibly with a single primary shard, assuming your retention period can handle that granularity. Aim for an average shard size around 10GB or larger.
---
### Author:  [@chenzilong444](https://discuss.elastic.co/u/chenzilong444)
#### Post date: [July 24, 2018, 9:03am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/10 "2018-07-24T09:03:19Z")
now i turn my heap to 3g and i change my index generation rule ，it generate quarterly.
and i close about 1200 shards,
heap.current heap.percent heap.max ram.max disk.avail node.role m
797.3mb 26 2.9gb 7.6gb 462.5gb mdi \*
now my problem is i can't keep them close.i want ask some way to reduce index or shards.
my index generate by logstash ,if i reindex,i can't imagine the workload
---
### Author:  [@chenzilong444](https://discuss.elastic.co/u/chenzilong444)
#### Post date: [July 24, 2018, 9:06am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/11 "2018-07-24T09:06:08Z")
and i can't find where to change the default shard number ,
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 21, 2018, 9:06am UTC](https://discuss.elastic.co/t/my-kibana-has-nothing-but-i-think-is-elasticsearch-problem/140661/12 "2018-08-21T09:06:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.