Hi,
i got this error when i tried to add some indices in AWS-ES
but this cluster currently has [1992]/[2000] maximum shards open
how can i solve this, if i want to keep that ES creates a delay index for each logs
Increase the number of nodes or reduce the number of shards per index.
What is the output of:
GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v
If some outputs are too big, please share them on gist.github.com and link them here.
BTW did you look at https://www.elastic.co/cloud and https://aws.amazon.com/marketplace/pp/B01N6YCISK ?
Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, APM, Logs UI, Infra UI, SIEM, Maps UI, AppSearch and what is coming next 
...
Hi @dadoonet
thanks for ur quick reply, these the output of the mentioned commands
GET /
{
"name" : "4ad0667b83bbe4b0b46d9c687fff253f",
"cluster_name" : "753188385451:geophy-logssystem",
"cluster_uuid" : "Dcyeu21UT1S7fL7weR1LBw",
"version" : {
"number" : "7.1.1",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "7a013de",
"build_date" : "2019-10-08T13:47:09.433683Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
GET /_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
x.x.x.x 5 81 1 0.00 0.17 0.41 mdi - 4ad0667b83bbe4b0b46d9c687fff253f
x.x.x.x 7 81 3 0.29 0.32 0.55 mdi * 542a9d72469d4dd9373b0099d18c4a2f
GET /_cat/health?v
epoch timestamp cluster status node.total node.data discovered_master shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1583325975 12:46:15 753188385451:xxxxxxxx green 2 2 true 2182 1091 0 0 0 0 - 100.0%for command GET /_cat/indices?v, all my indices in open status and green healthy
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open index1 XeOgZ3p3SF-VIBlL-OGB9A 5 1 5274 0 6.9mb 3.4mb
green open index2 Ynqo_kVyRaG_zhXgpf7I8w 5 1 102 0 273kb 139.6kb
green open evra-prod-alb-logs-2020.02.21 aMkKlQqEQOaGzt3KBBkhQg 5 1 23387 0 46.8mb 23.4mb
green open index3 8V-QPY4TRsyCJpVDbPAo5w 5 1 508 0 3.1mb 1.5mb
green open index4 fW6SQdbgQFO5wGwi0l1HGg 5 1 270 0 424.5kb 212.2kb
green open index5 VsTizVaLTU68ovVVEfQ93w 5 1 12708 0 16.9mb 8.4mb
green open index6 abMOPfiNTrCRZ68bvGe0eA 5 1 70703 0 92.6mb 46.3mb
green open index7 8oJz97tUQ5-cKdzOQq0vRQ 5 1 3361 0 3.3mb 1.6mb
green open index8 buH2H7lmSNOiLHdOvjFnoA 5 1 576 0 1mb 513.5kb
green open index9 4JPdqg62SEGHp9KUtbOYaA 5 1 7695 0 13.4mb 6.7mb
green open index10 WsOR1c8GTuKSERk4kkfwEQ 5 1 1187 0 2.3mb 1.1mb
Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.
Or use markdown style like:
```
CODE
```
This is the icon to use if you are not using markdown format:
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.
sorry @dadoonet, just edited
I highly doubt of this last output...
I can see 50 primary shards here where you have reported in total 1091 primaries.
@dadoonet
ys i didn't copy full output but all of them in started status
So I can just guess.
You probably have too many shards per node.
May I suggest you look at the following resources about sizing:
https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing
And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right
Thanks @dadoonet
sorry, how can i change the number of shared per index and is it recommended to change to monthly index instead of daily index ?
For existing indices the shrink API might be used.
For coming indices, create or modify the index template.
Monthly, yearly indices might be a good option. It depends on the index you have today. As you didn't want to share the full output, I can't really tell.
But you can also look at the rollover API.
sorrry @dadoonet. it's too long 
app2-prod-app-logs-2020.02.18 1 r STARTED 1534 587.8kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.18 1 p STARTED 1534 587.8kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-app-logs-2020.02.18 2 p STARTED 1463 570kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.18 2 r STARTED 1463 570kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-app-logs-2020.02.18 4 p STARTED 1452 565.6kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.18 4 r STARTED 1452 565.6kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-app-logs-2020.02.18 3 r STARTED 1500 574.6kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.18 3 p STARTED 1500 574.6kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-app-logs-2020.02.18 0 p STARTED 1474 582.1kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.18 0 r STARTED 1474 582.1kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-elb-access-log-2020.03.04 1 p STARTED 1683 1.8mb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-elb-access-log-2020.03.04 1 r STARTED 1683 1.7mb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-elb-access-log-2020.03.04 2 r STARTED 1725 1.8mb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-elb-access-log-2020.03.04 2 p STARTED 1725 1.9mb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-elb-access-log-2020.03.04 4 r STARTED 1717 2mb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-elb-access-log-2020.03.04 4 p STARTED 1717 2mb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-elb-access-log-2020.03.04 3 p STARTED 1644 1.8mb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-elb-access-log-2020.03.04 3 r STARTED 1644 1.7mb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-elb-access-log-2020.03.04 0 r STARTED 1743 2mb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-elb-access-log-2020.03.04 0 p STARTED 1743 2mb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
there is limit to past all there 
app2-prod-app-logs-2020.02.13 1 p STARTED 215 162.3kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.13 1 r STARTED 215 162.3kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-app-logs-2020.02.13 2 r STARTED 150 125.1kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.13 2 p STARTED 150 125.1kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-app-logs-2020.02.13 4 r STARTED 168 134.6kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.13 4 p STARTED 168 134.6kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-app-logs-2020.02.13 3 p STARTED 183 144.6kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.13 3 r STARTED 183 144.6kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-app-logs-2020.02.13 0 r STARTED 194 150.8kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-app-logs-2020.02.13 0 p STARTED 194 150.8kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app1-prod-alb-logs-2020.02.15 1 r STARTED 777 916.7kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app1-prod-alb-logs-2020.02.15 1 p STARTED 777 916.7kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app1-prod-alb-logs-2020.02.15 2 p STARTED 724 910.5kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app1-prod-alb-logs-2020.02.15 2 r STARTED 724 910.5kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app1-prod-alb-logs-2020.02.15 4 p STARTED 754 732.8kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app1-prod-alb-logs-2020.02.15 4 r STARTED 754 732.8kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app1-prod-alb-logs-2020.02.15 3 r STARTED 767 913.1kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app1-prod-alb-logs-2020.02.15 3 p STARTED 767 913.1kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app1-prod-alb-logs-2020.02.15 0 p STARTED 750 821.5kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app1-prod-alb-logs-2020.02.15 0 r STARTED 750 821.5kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-alb-logs-2020.02.16 1 r STARTED 796 742.3kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-alb-logs-2020.02.16 1 p STARTED 796 742.3kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-alb-logs-2020.02.16 2 p STARTED 769 823.8kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-alb-logs-2020.02.16 2 r STARTED 769 823.8kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-alb-logs-2020.02.16 4 p STARTED 737 820.6kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-alb-logs-2020.02.16 4 r STARTED 737 820.6kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-alb-logs-2020.02.16 3 r STARTED 778 752.8kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-alb-logs-2020.02.16 3 p STARTED 778 752.8kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
app2-prod-alb-logs-2020.02.16 0 p STARTED 725 729.9kb x.x.x.x 1688eee859e2418e813042d0b9f41f06
app2-prod-alb-logs-2020.02.16 0 r STARTED 725 729.9kb x.x.x.x f9d895cc2b7d1a07911e23dd1ae22959
You have so small indices that only one index with one single shard would be probably enough.
You can share the output on gist.github.com as well.
I'm more interested by the following than the /_cat/shards
GET /_cat/indices?v@dadoonet
so i should set 1 shared to each index, right?
With the size of your indices, I think that one single index with one single shard is enough.
@dadoonet
how can i set the default value for number of shared limit for coming indices. if i have different indices .
Also when i try to shrink the exist indices by shrink API "https://blog.ruanbekker.com/blog/2019/04/06/shrink-your-elasticsearch-index-by-reducing-the-shard-count-with-the-shards-api/", got the below error
"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[f9d895cc2b7d1a07911e23dd1ae22959][x.x.x.x:9300][indices:admin/create]"}],"type":"illegal_argument_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2172]/[2000] maximum shards open;"},"status":400}[ec2-user@ip-172-20-1-235 ~]
i need to create target index to archive the exist ones
You can may be temporary change the current limit with https://www.elastic.co/guide/en/elasticsearch/reference/7.6/misc-cluster.html#cluster-shard-limit
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.