# Metrics fail after several hours **URL:** \ **Category:** Metrics\ **Created:** [December 2, 2020, 9:21pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422 "2020-12-02T21:21:31Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![DucretJe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ducretje/32/79868_2.png) [@DucretJe](https://discuss.elastic.co/u/DucretJe)\ **Post date:** [December 2, 2020, 9:21pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/1 "2020-12-02T21:21:31Z")
**Elasticsearch** : Docker version 7.10 **Kibana** : Docker version 7.10 **Metricbeat** : Docker version 7.10 Hello, I have a strange issue with my ELK deployment. When I deploy Elasticsearch + Kibana, I get the metrics from the monitored server. The problem occurs several hours after the deployment (I don't know exactly but I'd say 10-12h) When I try to access to the metrics fronm the metric tab I get an "Internal Server Error (500)" URL http://:5601/api/metrics/snapshot When I want to see my dashboard ([Metricbeat System] Host overview ECS), I get a part of my metrics but I loose some of the panels, they display the following error: ``` The request for this panel failed all shards failed ``` The following panels are failing: - Inbound Traffic - Outbound Traffic - Disk Usage - Network Traffic (Packets) - Network Traffic (Bytes) - Processes By Memory - Processess By CPU - Interfaces By Incomming traffic - Interfaces by outgoing traffic I don't understand why it fails this way? ---
**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\ **Post date:** [December 2, 2020, 10:37pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/2 "2020-12-02T22:37:55Z")
Are there any errors in the Elasticsearch logs? My best guess is that when the index rolls over the mappings for the new indices are missing. That would explain why it starts out fine but then eventually starts failing after several hours. ---
**Author:** ![DucretJe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ducretje/32/79868_2.png) [@DucretJe](https://discuss.elastic.co/u/DucretJe)\ **Post date:** [December 2, 2020, 11:11pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/3 "2020-12-02T23:11:30Z")
Hi, Yes, I can see the stacktrace/ ``` "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:568) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:324) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:603) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:236) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:303) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] } {"type": "server", "timestamp": "2020-11-29T20:35:12,603+01:00", "level": "WARN", "component": "r.suppressed", "cluster.name": "docker-cluster", "node.name": "f6f831e1284f", "message": "path: /.kibana/_count, params: {i ndex=.kibana}", "cluster.uuid": "ncuITa4nTK-wbRWRg3_3Lg", "node.id": "g3e15271RR-DQMIQiSTk3g" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:568) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:324) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:603) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:236) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:303) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] } ``` I'm not very good with java... ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 2, 2020, 11:14pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/4 "2020-12-02T23:14:28Z")
Is there more to the log? ---
**Author:** ![DucretJe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ducretje/32/79868_2.png) [@DucretJe](https://discuss.elastic.co/u/DucretJe)\ **Post date:** [December 3, 2020, 8:30pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/5 "2020-12-03T20:30:41Z")
Here is a log from Kibana. Strange thing, no logs are generated by Elasticsearch?? [http://pastebin.fr/pastebin.php?dl=76496](http://pastebin.fr/pastebin.php?dl=76496) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 3, 2020, 9:34pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/6 "2020-12-03T21:34:01Z")
Can you please post a link to the text rather than a download. ---
**Author:** ![DucretJe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ducretje/32/79868_2.png) [@DucretJe](https://discuss.elastic.co/u/DucretJe)\ **Post date:** [December 3, 2020, 9:59pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/7 "2020-12-03T21:59:57Z")
of course > **[pastebin - outil de debug collaboratif](http://pastebin.fr/76520)** ---
**Author:** ![DucretJe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ducretje/32/79868_2.png) [@DucretJe](https://discuss.elastic.co/u/DucretJe)\ **Post date:** [December 6, 2020, 11:07am UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/8 "2020-12-06T11:07:27Z")
> [@simianhacker](#): > > My best guess is that when the index rolls over the mappings for the new indices are missing. That would explain why it starts out fine but then eventually starts failing after several hours. How could I check this? And how cold I fix it? ---
**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\ **Post date:** [December 7, 2020, 3:20pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/9 "2020-12-07T15:20:31Z")
Can you paste the output from ```auto GET metricbeat-*/_mapping ``` If you changed the pattern of your indices then adjust the query above. This will give us a listing of all the mappings for the indices queried for the Metrics UI. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 4, 2021, 3:20pm UTC](https://discuss.elastic.co/t/metrics-fail-after-several-hours/257422/10 "2021-01-04T15:20:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.