Error 503 on server post unblocking the index on a proxy

Elastic Stack Elasticsearch
1

Could not save: {'index': {'_index': 'prj_1a9c0440af7f4ed297736cffa5591082_connector', '_type': '_doc', '_id': 'model_d25q7003720200', 'status': 503, 'error': {'type': 'unavailable_shards_exception', 'reason': '[prj_1a9c0440af7f4ed297736cffa5591082_connector][4] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[prj_1a9c0440af7f4ed297736cffa5591082_connector][4]] containing [402] requests]'}}}

Need solution asap. thank you

2

Which version of Elasticsearch are you using?

What is the full output of the following APIs?

3

{
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 3,
"index" : "prj_7269ebdb1b97fc9e9d63ba79692350ac_assets",
"recovery_source" : {
"type" : "PEER"
},
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2020-12-04T15:41:47.021Z",
"failed_attempts" : 5,
"delayed" : false,
"details" : "failed shard on node [QhG_OTEpStehLVm8BKkzjw]: failed to create shard, failure AccessDeniedException[/usr/share/elasticsearch/data/nodes/0/indices/KmMVjuO7TqSec5cnapTspA/3]",
"allocation_status" : "no_attempt"
}
},
{
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 7,
"index" : "prj_7269ebdb1b97fc9e9d63ba79692350ac_assets",
"recovery_source" : {
"type" : "PEER"
},
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2020-12-04T15:41:46.812Z",
"failed_attempts" : 5,
"delayed" : false,
"details" : "failed shard on node [Yzmrlmh_RVSulSr3j9a_Ug]: failed to create shard, failure AccessDeniedException[/usr/share/elasticsearch/data/nodes/0/indices/KmMVjuO7TqSec5cnapTspA/7]",
"allocation_status" : "no_attempt"
}
},
{
"state" : "UNASSIGNED",
"primary" : false,
"node" : null,
"relocating_node" : null,
"shard" : 2,
"index" : "prj_7269ebdb1b97fc9e9d63ba79692350ac_assets",
"recovery_source" : {
"type" : "PEER"
},
"unassigned_info" : {
"reason" : "NODE_LEFT",
"at" : "2020-12-04T15:44:18.056Z",
"delayed" : false,
"details" : "node_left [Yzmrlmh_RVSulSr3j9a_Ug]",
"allocation_status" : "no_attempt"
}
},

This is the out put which i am getting and version is "number" : "6.8.0",

4

That is not the output of any of the APIs I asked for.

Are you using shared storage for Elasticsearch? What is the specification of your cluster?

5

Here you have it-
curl -XGET "http://127.0.0.1:9200/_cluster/stats"
{"_nodes":{"total":2,"successful":2,"failed":0},"cluster_name":"darksiders.cluster","cluster_uuid":"2u0u2QbrS426ZyZqN_m_4w","timestamp":1607154046320,"status":"red","indices":{"count":5,"shards":{"total":12,"primaries":12,"replication":0.0,"index":{"shards":{"min":1,"max":4,"avg":2.4},"primaries":{"min":1,"max":4,"avg":2.4},"replication":{"min":0.0,"max":0.0,"avg":0.0}}},"docs":{"count":526516,"deleted":140228},"store":{"size_in_bytes":1886904179},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"query_cache":{"memory_size_in_bytes":9642744,"total_count":11453,"hit_count":4027,"miss_count":7426,"cache_size":565,"cache_count":785,"evictions":220},"completion":{"size_in_bytes":0},"segments":{"count":101,"memory_in_bytes":11624034,"terms_memory_in_bytes":9338581,"stored_fields_memory_in_bytes":136624,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":1480640,"points_memory_in_bytes":404673,"doc_values_memory_in_bytes":263516,"index_writer_memory_in_bytes":0,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":87224,"max_unsafe_auto_id_timestamp":-1,"file_sizes":{}}},"nodes":{"count":{"total":2,"data":1,"coordinating_only":0,"master":1,"ingest":1},"versions":["6.8.0"],"os":{"available_processors":48,"allocated_processors":48,"names":[{"name":"Linux","count":2}],"pretty_names":[{"pretty_name":"CentOS Linux 7 (Core)","count":2}],"mem":{"total_in_bytes":134746841088,"free_in_bytes":44399255552,"used_in_bytes":90347585536,"free_percent":33,"used_percent":67}},"process":{"cpu":{"percent":0},"open_file_descriptors":{"min":493,"max":614,"avg":553}},"jvm":{"max_uptime_in_millis":13477550653,"versions":[{"version":"12.0.1","vm_name":"OpenJDK 64-Bit Server VM","vm_version":"12.0.1+12","vm_vendor":"Oracle Corporation","count":2}],"mem":{"heap_used_in_bytes":2664205872,"heap_max_in_bytes":4151836672},"threads":371},"fs":{"total_in_bytes":1181231001600,"free_in_bytes":824034258944,"available_in_bytes":764007268352},"plugins":,"network_types":{"transport_types":{"security4":2},"http_types":{"security4":2}}}}

And this:
curl -XGET "http://127.0.0.1:9200/_cluster/health"
{"cluster_name":"darksiders.cluster","status":"red","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":1,"active_primary_shards":12,"active_shards":12,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":168,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":6.666666666666667}

6

It would help if you could tell us a bit more about the cluster and the hardware/storage used.

This to me sounds like a potential file system problem.

7

Hardware:

CPU cores: 24
RAM:64Gb
HDD: 1Tb

and yes its a shared storage for ES , couchdb and postgres all this runs using docker container.

8

There was no issue until 2 days back. If you can suggest anything to unblock this situation it would be really helpful.

Yesterday one of the linked proxy had full disk storage so post increasing the space i just unblocked indexes using command - curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

But this issue is happening on the server afterwards.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.