# Queuing tasks and snapshot operation not working **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring\ **Created:** [December 10, 2020, 2:19pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286 "2020-12-10T14:19:30Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![afonsoavr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/afonsoavr/32/65059_2.png) [@afonsoavr](https://discuss.elastic.co/u/afonsoavr)\ **Post date:** [December 10, 2020, 2:19pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/1 "2020-12-10T14:19:31Z")
Hi, I has the cluster elasticsearch with 6 data nodes and 3 master. When execute the snapshot I receive the error " **process\_cluster\_event\_timeout\_exception**". I look in my cluster " **/\_cat/pending\_tasks**" it has 69 tasks with _priority_ **HIGH** and _source_ **put-mapping** My cluster is for centralized log and have this process to put data in cluster: - logstash - collect from Redis and put to Elasticsearch - apm-server - filebeat - metricbeat I stay removing beats and some applications from apm-server Has as change **priority** for create\_snapshot from _NORMAL_ to _HIGH_? It is not a solution, how to I check the correct size for my cluster? \*Normally i keep 7 days the indice in my cluster because the backup. But because the error, I removed the process to exclude the old data _GET \_cat/nodes?v&s=node.role:desc_ | ip | heap.percent | ram.percent | cpu | load\_1m | load\_5m | load\_15m | node.role | master | name | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 10.0.2.8 | 47 | 50 | 0 | 0.00 | 0.00 | 0.00 | mi | - | prd-elasticsearch-i-020 | | 10.0.0.7 | 14 | 50 | 0 | 0.00 | 0.00 | 0.00 | mi | - | prd-elasticsearch-i-0ab | | 10.0.1.1 | 47 | 77 | 29 | 1.47 | 1.72 | 1.66 | mi | \* | prd-elasticsearch-i-0e2 | | 10.0.2.7 | 58 | 95 | 19 | 8.04 | 8.62 | 8.79 | d | - | prd-elasticsearch-i-0b4 | | 10.0.2.4 | 59 | 97 | 20 | 8.22 | 8.71 | 8.76 | d | - | prd-elasticsearch-i-00d | | 10.0.1.6 | 62 | 94 | 38 | 11.42 | 8.87 | 8.89 | d | - | prd-elasticsearch-i-0ff | | 10.0.0.6 | 67 | 97 | 25 | 8.97 | 10.45 | 10.47 | d | - | prd-elasticsearch-i-01a | | 10.0.0.9 | 57 | 98 | 32 | 11.63 | 9.64 | 9.17 | d | - | prd-elasticsearch-i-005 | | 10.0.1.0 | 62 | 96 | 19 | 10.45 | 9.53 | 9.31 | d | - | prd-elasticsearch-i-088 | **My cluster definitions:** ```json { "_nodes": { "total": 9, "successful": 9, "failed": 0 }, "cluster_name": "prd-elasticsearch", "cluster_uuid": "xxxx", "timestamp": 1607609607018, "status": "green", "indices": { "count": 895, "shards": { "total": 14006, "primaries": 4700, "replication": 1.98, "index": { "shards": { "min": 2, "max": 18, "avg": 15.649162011173184 }, "primaries": { "min": 1, "max": 6, "avg": 5.251396648044692 }, "replication": { "min": 1, "max": 2, "avg": 1.9787709497206705 } } }, "docs": { "count": 14896803950, "deleted": 843126 }, "store": { "size_in_bytes": 16778620001453 }, "fielddata": { "memory_size_in_bytes": 4790672272, "evictions": 0 }, "query_cache": { "memory_size_in_bytes": 7689832903, "total_count": 2033762560, "hit_count": 53751516, "miss_count": 1980011044, "cache_size": 4087727, "cache_count": 11319866, "evictions": 7232139 }, "completion": { "size_in_bytes": 0 }, "segments": { "count": 155344, "memory_in_bytes": 39094918196, "terms_memory_in_bytes": 31533157295, "stored_fields_memory_in_bytes": 5574613712, "term_vectors_memory_in_bytes": 0, "norms_memory_in_bytes": 449973760, "points_memory_in_bytes": 886771949, "doc_values_memory_in_bytes": 650401480, "index_writer_memory_in_bytes": 905283962, "version_map_memory_in_bytes": 1173400, "fixed_bit_set_memory_in_bytes": 12580800, "max_unsafe_auto_id_timestamp": 1607606224903, "file_sizes": {} } }, "nodes": { "count": { "total": 9, "data": 6, "coordinating_only": 0, "master": 3, "ingest": 3 }, "versions": [ "6.8.1" ], "os": { "available_processors": 108, "allocated_processors": 108, "names": [ { "name": "Linux", "count": 9 } ], "pretty_names": [ { "pretty_name": "CentOS Linux 7 (Core)", "count": 9 } ], "mem": { "total_in_bytes": 821975162880, "free_in_bytes": 50684043264, "used_in_bytes": 771291119616, "free_percent": 6, "used_percent": 94 } }, "process": { "cpu": { "percent": 349 }, "open_file_descriptors": { "min": 429, "max": 9996, "avg": 6607 } }, "jvm": { "max_uptime_in_millis": 43603531934, "versions": [ { "version": "1.8.0_222", "vm_name": "OpenJDK 64-Bit Server VM", "vm_version": "25.222-b10", "vm_vendor": "Oracle Corporation", "count": 9 } ], "mem": { "heap_used_in_bytes": 137629451248, "heap_max_in_bytes": 205373571072 }, "threads": 1941 }, "fs": { "total_in_bytes": 45245361229824, "free_in_bytes": 28231010959360, "available_in_bytes": 28231011147776 }, "plugins": [ { "name": "repository-s3", "version": "6.8.1", "elasticsearch_version": "6.8.1", "java_version": "1.8", "description": "The S3 repository plugin adds S3 repositories", "classname": "org.elasticsearch.repositories.s3.S3RepositoryPlugin", "extended_plugins": [], "has_native_controller": false } ], "network_types": { "transport_types": { "security4": 9 }, "http_types": { "security4": 9 } } } } ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 10, 2020, 2:26pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/2 "2020-12-10T14:26:42Z")
> [@afonsoavr](#): > > I look in my cluster " **/\_cat/pending\_tasks**" it has 69 tasks with _priority_ **HIGH** and _source_ **put-mapping** This sounds like the problem, you're updating mappings too often. The first thing I'd suggest is to stop doing that. ---
**Author:** ![afonsoavr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/afonsoavr/32/65059_2.png) [@afonsoavr](https://discuss.elastic.co/u/afonsoavr)\ **Post date:** [December 10, 2020, 2:47pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/3 "2020-12-10T14:47:07Z")
> [@DavidTurner](#): > > This sounds like the problem, you're updating mappings too often. The first thing I'd suggest is to stop doing that. Hi @DavidTurner, thank's for your update. How to stop or minimize the process put-mapping? I using logstash, metricbeat, filebeat to put data. How to identify the process cause the put-mapping? For example: _GET /\_tasks?detailed=true_ ```auto "1EWDKShaR4muQ2_6KCd2dg:1059431559" : { "node" : "1EWDKShaR4muQ2_6KCd2dg", "id" : 1059431559, "type" : "netty", "action" : "indices:admin/mapping/put", "description" : "", "start_time_in_millis" : 1607611398514, "running_time_in_nanos" : 608622968, "cancellable" : false, "parent_task_id" : "y0uMNye4Sc-y5lM0p59G6Q:513942027", "headers" : { } }, "1EWDKShaR4muQ2_6KCd2dg:1059431558" : { "node" : "1EWDKShaR4muQ2_6KCd2dg", "id" : 1059431558, "type" : "netty", "action" : "indices:admin/mapping/put", "description" : "", "start_time_in_millis" : 1607611398498, "running_time_in_nanos" : 624527171, "cancellable" : false, "parent_task_id" : "Ny8ihNH2SO-qxXROkZj4-A:575706969", "headers" : { } }, ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 10, 2020, 3:03pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/4 "2020-12-10T15:03:05Z")
Good question. Check `GET /_cluster/pending_tasks?pretty&human`, I think this identifies the index whose mapping is being updated. ---
**Author:** ![afonsoavr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/afonsoavr/32/65059_2.png) [@afonsoavr](https://discuss.elastic.co/u/afonsoavr)\ **Post date:** [December 10, 2020, 3:21pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/5 "2020-12-10T15:21:17Z")
Hi @DavidTurner, thank's for your update, For this command I receive that result: ```json { "tasks" : [ { "insert_order" : 268970200, "priority" : "HIGH", "source" : "put-mapping", "executing" : true, "time_in_queue_millis" : 1453, "time_in_queue" : "1.4s" }, { "insert_order" : 268970201, "priority" : "HIGH", "source" : "put-mapping", "executing" : false, "time_in_queue_millis" : 1451, "time_in_queue" : "1.4s" }, .... ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 10, 2020, 3:33pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/6 "2020-12-10T15:33:34Z")
Oh, that's not helpful. What version is this? ---
**Author:** ![afonsoavr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/afonsoavr/32/65059_2.png) [@afonsoavr](https://discuss.elastic.co/u/afonsoavr)\ **Post date:** [December 10, 2020, 3:36pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/7 "2020-12-10T15:36:12Z")
Hi, @DavidTurner, My version is 6.8.1 > [@afonsoavr](#): > > ```auto > "versions": [ > "6.8.1" > ], > > "jvm": { > "max_uptime_in_millis": 43603531934, > "versions": [ > { > "version": "1.8.0_222", > "vm_name": "OpenJDK 64-Bit Server VM", > "vm_version": "25.222-b10", > "vm_vendor": "Oracle Corporation", > "count": 9 > } > ], > "mem": { > "heap_used_in_bytes": 137629451248, > "heap_max_in_bytes": 205373571072 > }, > "threads": 1941 > }, > > ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 10, 2020, 4:35pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/8 "2020-12-10T16:35:10Z")
Ah ok we only added the index name to the output in [https://github.com/elastic/elasticsearch/pull/52690](https://github.com/elastic/elasticsearch/pull/52690) (i.e. 7.7.0). I don't have any other great suggestions for such an old version as 6.8.1, sorry. It could well be any of the clients you listed. ---
**Author:** ![afonsoavr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/afonsoavr/32/65059_2.png) [@afonsoavr](https://discuss.elastic.co/u/afonsoavr)\ **Post date:** [December 10, 2020, 5:02pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/9 "2020-12-10T17:02:17Z")
Understand this possible. Thank's ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 7, 2021, 5:02pm UTC](https://discuss.elastic.co/t/queuing-tasks-and-snapshot-operation-not-working/258286/10 "2021-01-07T17:02:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.