# ERROR - collector \[index-stats\] timed out when collecting data

**URL:** <https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353>\
**Category:** Вопросы на русском языке\
**Created:** [August 18, 2022, 7:21am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353 "2022-08-18T07:21:30Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![AntonUstinov](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@AntonUstinov](https://discuss.elastic.co/u/AntonUstinov)\
**Post date:** [August 18, 2022, 7:21am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/1 "2022-08-18T07:21:30Z")

</div>

Регулярно стали появляться такие ошибки в логах, после чего эластик (6.8) вылетает. Этот вопрос уже несколько раз поднимался в англоязычном сегменте, но подходящих ответов я там не нашел (писали, что могут быть проблемы с железом или что используется слишком много шард). Может здесь кто-нибудь подскажет, где искать проблему?

 ![el-monitor](https://us1.discourse-cdn.com/elastic/original/3X/b/3/b3fb778d211765ef0f22f692373303cbf7c29e87.png)

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [August 18, 2022, 8:48am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/2 "2022-08-18T08:48:41Z")

</div>

It looks like your service was stoped for some time. Is it regular, for example every 24 hours?

---

<div class="post-metadata">

**Author:** ![AntonUstinov](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@AntonUstinov](https://discuss.elastic.co/u/AntonUstinov)\
**Post date:** [August 18, 2022, 9:07am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/3 "2022-08-18T09:07:49Z")

</div>

Yes, I wrote a bash script that checks the status of elasticsearch service and restarts it when it is inactive. This happens regularly, but not every 24 hours, sometimes with a break of several days.

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [August 18, 2022, 9:12am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/4 "2022-08-18T09:12:03Z")

</div>

Could you please paste your config file (with \</\> tag)? Of course please cover your sensitive data.

---

<div class="post-metadata">

**Author:** ![AntonUstinov](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@AntonUstinov](https://discuss.elastic.co/u/AntonUstinov)\
**Post date:** [August 18, 2022, 9:21am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/5 "2022-08-18T09:21:41Z")

</div>

```auto
cluster.name: ***
node.name: master-node
node.master: true

path:
  data:
    - /mnt/elasticsearch_1
    - /mnt/elasticsearch_2
    - /mnt/elasticsearch_3

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true
network.bind_host: [localhost, *.*.*.*]
xpack.security.enabled: false
path.repo: ["/mnt/snapshot"]

```

---

<div class="post-metadata">

**Author:** ![AntonUstinov](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@AntonUstinov](https://discuss.elastic.co/u/AntonUstinov)\
**Post date:** [August 18, 2022, 9:24am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/6 "2022-08-18T09:24:43Z")

</div>

```auto
## JVM configuration
-Xms2g
-Xmx2g

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10

## optimizations
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
14-:-XX:+ShowCodeDetailsInExceptionMessages
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j2.formatMsgNoLookups=true
-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

## JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=8
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT

# temporary workaround for C2 bug with JDK 10 on hardware with AVX-512
10-:-XX:UseAVX=2

```

---

<div class="post-metadata">

**Author:** ![AntonUstinov](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@AntonUstinov](https://discuss.elastic.co/u/AntonUstinov)\
**Post date:** [August 18, 2022, 9:29am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/7 "2022-08-18T09:29:35Z")

</div>

![elastic-info](https://us1.discourse-cdn.com/elastic/original/3X/e/a/eaa7d471da9ab1173b6c2a5b406b85d4c0e93229.png)

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [August 18, 2022, 9:38am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/8 "2022-08-18T09:38:28Z")

</div>

All look good.  
And when it happens does your log say anything? Maybe system journal?

---

<div class="post-metadata">

**Author:** ![AntonUstinov](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@AntonUstinov](https://discuss.elastic.co/u/AntonUstinov)\
**Post date:** [August 18, 2022, 10:01am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/9 "2022-08-18T10:01:36Z")

</div>

There are no entries in the system journal during this period...

Elasticsearch logs:

```auto
[2022-08-17T00:00:01,323][INFO][o.e.c.m.MetaDataIndexTemplateService] [master-node] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
[2022-08-17T01:00:00,000][INFO][o.e.x.m.e.l.LocalExporter] [master-node] cleaning up [2] old indices
[2022-08-17T01:00:00,006][INFO][o.e.c.m.MetaDataDeleteIndexService] [master-node] [.monitoring-es-6-2022.08.09/ab5DMOIlTUizH59sHLNsqA] deleting index
[2022-08-17T01:00:00,006][INFO][o.e.c.m.MetaDataDeleteIndexService] [master-node] [.monitoring-kibana-6-2022.08.09/dEZ1Uxk7T3KfBvUZBLhAYg] deleting index
[2022-08-17T02:09:29,350][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-node] collector [index-stats] timed out when collecting data
[2022-08-17T02:10:06,837][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:10:36,838][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:10:56,839][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:11:06,959][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-node] collector [index-stats] timed out when collecting data
[2022-08-17T02:11:17,069][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [master-node] collector [cluster_stats] timed out when collecting data
[2022-08-17T02:11:36,839][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:11:46,945][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-node] collector [index-stats] timed out when collecting data
[2022-08-17T02:11:57,069][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [master-node] collector [cluster_stats] timed out when collecting data
[2022-08-17T02:12:16,840][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:12:26,974][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-node] collector [index-stats] timed out when collecting data
[2022-08-17T02:12:46,841][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:12:56,824][WARN][o.e.c.InternalClusterInfoService] [master-node] Failed to update shard information for ClusterInfoUpdateJob within 15s timeout
[2022-08-17T02:12:56,998][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-node] collector [index-stats] timed out when collecting data
[2022-08-17T02:13:16,842][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:13:26,961][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-node] collector [index-stats] timed out when collecting data
[2022-08-17T02:13:46,842][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:13:56,967][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-node] collector [index-stats] timed out when collecting data
[2022-08-17T02:14:07,108][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [master-node] collector [cluster_stats] timed out when collecting data
[2022-08-17T02:14:26,843][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:14:40,068][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-node] collector [index-stats] timed out when collecting data
[2022-08-17T02:14:50,568][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [master-node] collector [cluster_stats] timed out when collecting data
[2022-08-17T02:15:00,000][INFO][o.e.x.m.MlDailyMaintenanceService] [master-node] triggering scheduled [ML] maintenance tasks
[2022-08-17T02:15:00,326][INFO][o.e.x.m.a.TransportDeleteExpiredDataAction] [master-node] Deleting expired data
[2022-08-17T02:15:00,331][INFO][o.e.x.m.a.TransportDeleteExpiredDataAction] [master-node] Completed deletion of expired ML data
[2022-08-17T02:15:00,331][INFO][o.e.x.m.MlDailyMaintenanceService] [master-node] Successfully completed [ML] maintenance tasks
[2022-08-17T02:15:06,844][ERROR][o.e.x.m.c.n.NodeStatsCollector] [master-node] collector [node_stats] timed out when collecting data
[2022-08-17T02:15:20,408][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-node] collector [index-stats] timed out when collecting data
[2022-08-17T02:15:31,112][INFO][o.e.e.NodeEnvironment] [master-node] using [3] data paths, mounts [[/ (/dev/vda2)]], net usable_space [82gb], net total_space [177.1gb], types [ext4]
[2022-08-17T02:15:31,116][INFO][o.e.e.NodeEnvironment] [master-node] heap size [2gb], compressed ordinary object pointers [true]
[2022-08-17T02:15:31,648][INFO][o.e.n.Node] [master-node] node name [master-node], node ID [WbYUANt3SeOJaH9JNP3Gsw]
[2022-08-17T02:15:31,649][INFO][o.e.n.Node] [master-node] version[6.8.23], pid[2932703], build[default/deb/4f67856/2022-01-06T21:30:50.087716Z], OS[Linux/5.4.0-105-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/17.0.2/17.0.2+8-86]
[2022-08-17T02:15:31,649][INFO][o.e.n.Node] [master-node] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.io.tmpdir=/tmp/elasticsearch-14765442649856788009, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb]
[2022-08-17T02:15:34,448][INFO][o.e.p.PluginsService] [master-node] loaded module [aggs-matrix-stats]
[2022-08-17T02:15:34,448][INFO][o.e.p.PluginsService] [master-node] loaded module [analysis-common]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [ingest-common]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [ingest-geoip]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [ingest-user-agent]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [lang-expression]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [lang-mustache]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [lang-painless]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [mapper-extras]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [parent-join]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [percolator]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [rank-eval]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [reindex]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [repository-url]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [transport-netty4]
[2022-08-17T02:15:34,449][INFO][o.e.p.PluginsService] [master-node] loaded module [tribe]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-ccr]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-core]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-deprecation]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-graph]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-ilm]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-logstash]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-ml]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-monitoring]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-rollup]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-security]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-sql]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-upgrade]
[2022-08-17T02:15:34,450][INFO][o.e.p.PluginsService] [master-node] loaded module [x-pack-watcher]
[2022-08-17T02:15:34,451][INFO][o.e.p.PluginsService] [master-node] no plugins loaded
[2022-08-17T02:15:38,531][INFO][o.e.x.m.p.l.CppLogMessageHandler] [master-node] [controller/2933091] [Main.cc@114] controller (64 bit): Version 6.8.23 (Build 31256deab94add) Copyright (c) 2022 Elasticsearch BV
[2022-08-17T02:15:39,337][INFO][o.e.d.DiscoveryModule] [master-node] using discovery type [zen] and host providers [settings]
[2022-08-17T02:15:39,945][INFO][o.e.n.Node] [master-node] initialized
[2022-08-17T02:15:39,945][INFO][o.e.n.Node] [master-node] starting ...

```

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [August 18, 2022, 10:33am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/10 "2022-08-18T10:33:30Z")

</div>

> [@AntonUstinov](#):
>
> `[master-node] collector [node_stats] timed out when collecting data`

I've found this: [link\_1](https://discuss.elastic.co/t/trying-to-understand-timed-out-when-collecting-data/113979/7) and this: [link\_2](https://discuss.elastic.co/t/error-o-e-x-m-c-n-nodestatscollector-collector-node-stats-timed-out-when-collecting-data/244194/6)

Have you checked your specifications and compared it to ES needs?

---

<div class="post-metadata">

**Author:** ![AntonUstinov](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@AntonUstinov](https://discuss.elastic.co/u/AntonUstinov)\
**Post date:** [August 18, 2022, 11:35am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/11 "2022-08-18T11:35:03Z")

</div>

Thank you. Although I have already looked at these posts, it gave me the idea of exceeding the limits on disk read / write operations.

 ![disk-read](https://us1.discourse-cdn.com/elastic/original/3X/b/e/be5d954fd022bbf032ee100560387f8d559e2f86.png)  
The max read/write bandwidth is 90 MBps.

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [August 18, 2022, 12:04pm UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/12 "2022-08-18T12:04:08Z")

</div>

It looks like you have found the issue. Please let me know if changing limits made your system work properly.

---

<div class="post-metadata">

**Author:** ![AntonUstinov](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@AntonUstinov](https://discuss.elastic.co/u/AntonUstinov)\
**Post date:** [August 28, 2022, 8:32am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/13 "2022-08-28T08:32:58Z")

</div>

Now I'm pretty sure that in my case the problem was disk overload during heavy indexing with 'upserts'. I've tweaked the index update logic a bit and everything seems to be working fine now. At the moment, Elasticsearch's uptime is around 7 days.

 ![disk read](https://us1.discourse-cdn.com/elastic/original/3X/1/1/11f62afc9cb886ebe29d2e2d74f645cfad1fedb3.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 25, 2022, 8:33am UTC](https://discuss.elastic.co/t/error-collector-index-stats-timed-out-when-collecting-data/312353/14 "2022-09-25T08:33:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.