# Service unavailable error code 503 all shard failed
**URL:** https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976
**Category:** Elasticsearch
**Created:** [May 22, 2023, 6:54am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976 "2023-05-22T06:54:28Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@target\_test](https://discuss.elastic.co/u/target_test)
#### Post date: [May 22, 2023, 6:54am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/1 "2023-05-22T06:54:28Z")
Hello all,
I got this problem showing that service unavailable
```auto
{"statusCode":503,"error":"Service Unavailable","message":"[all shards failed: search_phase_execution_exception\n\tRoot causes:\n\t\tno_shard_available_action_exception: null]: all shards failed"}
```
i am new with this kinda stuff. I got this problem after refreshing discover tab
this is what i get when using get \_cat/shards?v
```auto
.ds-logs-elastic_agent.metricbeat-default-2023.05.14-000005 0 p STARTED 52805 13.2mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.metricbeat-default-2023.05.14-000005 0 r UNASSIGNED
.async-search 0 p STARTED 0 4kb 127.0.0.1 xenpher
.ds-logs-system.application-default-2023.05.17-000007 0 p STARTED 259 463.9kb 127.0.0.1 xenpher
.ds-logs-system.application-default-2023.05.17-000007 0 r UNASSIGNED
.ds-logs-elastic_agent.filebeat-default-2023.05.14-000005 0 p STARTED 21642 7.3mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.14-000005 0 r UNASSIGNED
.ds-logs-endpoint.events.library-default-2023.05.15-000005 0 p STARTED 3845 1.9mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.library-default-2023.05.15-000005 0 r UNASSIGNED
.kibana-event-log-8.7.1-000001 0 p STARTED 29823516 4.6gb 127.0.0.1 xenpher
.ds-metrics-system.process-default-2023.05.04-000001 0 p STARTED 3390526 2.3gb 127.0.0.1 xenpher
.ds-metrics-system.process-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.registry-default-2023.05.13-000003 0 p STARTED 29906 6mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.registry-default-2023.05.13-000003 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.19-000010 0 p STARTED 20763 4.4mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.19-000010 0 r UNASSIGNED
.ds-logs-endpoint.events.process-default-2023.05.19-000009 0 p STARTED 1993780 850mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.19-000009 0 r UNASSIGNED
.ds-logs-endpoint.alerts-default-2023.05.19-000004 0 p STARTED 0 225b 127.0.0.1 xenpher
.ds-logs-endpoint.alerts-default-2023.05.19-000004 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.14-000005 0 p STARTED 48967 7.3mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.14-000005 0 r UNASSIGNED
.ds-logs-endpoint.alerts-default-2023.05.12-000002 0 p STARTED 20 214.7kb 127.0.0.1 xenpher
.ds-logs-endpoint.alerts-default-2023.05.12-000002 0 r UNASSIGNED
filebeat-1 0 p STARTED 0 225b 127.0.0.1 xenpher
filebeat-1 0 r UNASSIGNED
.fleet-file-data-agent-000001 0 p STARTED 0 225b 127.0.0.1 xenpher
.ds-metrics-system.load-default-2023.05.04-000001 0 p STARTED 289403 65.2mb 127.0.0.1 xenpher
.ds-metrics-system.load-default-2023.05.04-000001 0 r UNASSIGNED
.lists-default-000001 0 p STARTED 0 225b 127.0.0.1 xenpher
.lists-default-000001 0 r UNASSIGNED
.ds-logs-elastic_agent.fleet_server-default-2023.05.12-000003 0 p STARTED 899 1.6mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.fleet_server-default-2023.05.12-000003 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.16-000007 0 p STARTED 49202 7mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.16-000007 0 r UNASSIGNED
.ds-logs-system.application-default-2023.05.13-000003 0 p STARTED 67 164.1kb 127.0.0.1 xenpher
.ds-logs-system.application-default-2023.05.13-000003 0 r UNASSIGNED
.fleet-files-endpoint-000001 0 p STARTED 0 225b 127.0.0.1 xenpher
.ds-metrics-elastic_agent.filebeat-default-2023.05.04-000001 0 p STARTED 817728 173.5mb 127.0.0.1 xenpher
.ds-metrics-elastic_agent.filebeat-default-2023.05.04-000001 0 r UNASSIGNED
.security-7 0 p STARTED 921 603kb 127.0.0.1 xenpher
.apm-custom-link 0 p STARTED 0 225b 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.14-000004 0 p STARTED 195564 34.6mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.14-000004 0 r UNASSIGNED
.ds-metrics-endpoint.metrics-default-2023.05.04-000001 0 p STARTED 123 986.7kb 127.0.0.1 xenpher
.ds-metrics-endpoint.metrics-default-2023.05.04-000001 0 r UNASSIGNED
.apm-source-map 0 p STARTED 0 225b 127.0.0.1 xenpher
.ds-logs-elastic_agent.fleet_server-default-2023.05.16-000007 0 p STARTED 449 360.7kb 127.0.0.1 xenpher
.ds-logs-elastic_agent.fleet_server-default-2023.05.16-000007 0 r UNASSIGNED
.ds-metrics-system.process.summary-default-2023.05.04-000001 0 p STARTED 413360 82.2mb 127.0.0.1 xenpher
.ds-metrics-system.process.summary-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.process-default-2023.05.22-000013 0 p STARTED 3974605 1.6gb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.22-000013 0 r UNASSIGNED
.ds-metrics-system.network-default-2023.05.04-000001 0 p STARTED 1395798 207.3mb 127.0.0.1 xenpher
.ds-metrics-system.network-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.network-default-2023.05.04-000001 0 p STARTED 310644 100.7mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.network-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-system.system-default-2023.05.12-000002 0 p STARTED 57 304.2kb 127.0.0.1 xenpher
.ds-logs-system.system-default-2023.05.12-000002 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.13-000004 0 p STARTED 47040 5.1mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.13-000004 0 r UNASSIGNED
.ds-logs-elastic_agent.fleet_server-default-2023.05.14-000005 0 p STARTED 901 1.6mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.fleet_server-default-2023.05.14-000005 0 r UNASSIGNED
.ds-logs-endpoint.events.security-default-2023.05.13-000003 0 p STARTED 640 590.4kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.13-000003 0 r UNASSIGNED
.ds-logs-system.application-default-2023.05.15-000005 0 p STARTED 152 486.5kb 127.0.0.1 xenpher
.ds-logs-system.application-default-2023.05.15-000005 0 r UNASSIGNED
.ds-logs-endpoint.events.security-default-2023.05.19-000008 0 p STARTED 572 464.8kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.19-000008 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.12-000002 0 p STARTED 97677 22.2mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.12-000002 0 r UNASSIGNED
.ds-logs-endpoint.events.api-default-2023.05.12-000002 0 p STARTED 4 57.7kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.api-default-2023.05.12-000002 0 r UNASSIGNED
.ds-logs-system.security-default-2023.05.16-000006 0 p STARTED 4109 1.5mb 127.0.0.1 xenpher
.ds-logs-system.security-default-2023.05.16-000006 0 r UNASSIGNED
.ds-logs-elastic_agent-default-2023.05.04-000001 0 p STARTED 1638 639.7kb 127.0.0.1 xenpher
.ds-logs-elastic_agent-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-elastic_agent.filebeat-default-2023.05.17-000008 0 p STARTED 90329 20mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.17-000008 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.13-000003 0 p STARTED 97494 21.2mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.13-000003 0 r UNASSIGNED
.internal.alerts-security.alerts-default-000001 0 p STARTED 6108 26.8mb 127.0.0.1 xenpher
.ds-logs-system.auth-default-2023.05.12-000002 0 p STARTED 97 221.6kb 127.0.0.1 xenpher
.ds-logs-system.auth-default-2023.05.12-000002 0 r UNASSIGNED
.ds-logs-endpoint.events.security-default-2023.05.04-000001 0 p STARTED 5713 1.3mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.library-default-2023.05.13-000003 0 p STARTED 7828 2.4mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.library-default-2023.05.13-000003 0 r UNASSIGNED
.ds-logs-elastic_agent.fleet_server-default-2023.05.05-000002 0 p STARTED 4784 7.5mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.fleet_server-default-2023.05.05-000002 0 r UNASSIGNED
.ds-logs-elastic_agent.fleet_server-default-2023.05.17-000008 0 p STARTED 3884 6.7mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.fleet_server-default-2023.05.17-000008 0 r UNASSIGNED
.ds-metrics-system.cpu-default-2023.05.04-000001 0 p STARTED 413442 78.2mb 127.0.0.1 xenpher
.ds-metrics-system.cpu-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.registry-default-2023.05.14-000004 0 p STARTED 89224 14.2mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.registry-default-2023.05.14-000004 0 r UNASSIGNED
.ds-metrics-system.diskio-default-2023.05.04-000001 0 p STARTED 6183433 914.5mb 127.0.0.1 xenpher
.ds-metrics-system.diskio-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-system.application-default-2023.05.04-000001 0 p STARTED 8187 2.6mb 127.0.0.1 xenpher
.ds-logs-system.application-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-elastic_agent-default-2023.05.14-000005 0 p STARTED 273 374.8kb 127.0.0.1 xenpher
.ds-logs-elastic_agent-default-2023.05.14-000005 0 r UNASSIGNED
.ds-metrics-system.socket_summary-default-2023.05.04-000001 0 p STARTED 413415 96.5mb 127.0.0.1 xenpher
.ds-metrics-system.socket_summary-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.api-default-2023.05.16-000004 0 p STARTED 4 57.7kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.api-default-2023.05.16-000004 0 r UNASSIGNED
.ds-logs-system.auth-default-2023.05.13-000003 0 p STARTED 98 284.7kb 127.0.0.1 xenpher
.ds-logs-system.auth-default-2023.05.13-000003 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.18-000009 0 p STARTED 16242 3.2mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.18-000009 0 r UNASSIGNED
.kibana_security_session_1 0 p STARTED 1 6.6kb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.05-000002 0 p STARTED 259405 26.8mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.05-000002 0 r UNASSIGNED
.ds-logs-endpoint.events.library-default-2023.05.04-000001 0 p STARTED 81395 15.5mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.library-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.registry-default-2023.05.15-000005 0 p STARTED 49085 8mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.registry-default-2023.05.15-000005 0 r UNASSIGNED
.ds-logs-endpoint.events.registry-default-2023.05.04-000001 0 p STARTED 458053 67.3mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.registry-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-elastic_agent-default-2023.05.12-000003 0 p STARTED 144 303.8kb 127.0.0.1 xenpher
.ds-logs-elastic_agent-default-2023.05.12-000003 0 r UNASSIGNED
.ds-logs-endpoint.events.network-default-2023.05.16-000006 0 p STARTED 14427 4.4mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.network-default-2023.05.16-000006 0 r UNASSIGNED
.ds-logs-system.syslog-default-2023.05.17-000007 0 p STARTED 4361071 707.6mb 127.0.0.1 xenpher
.ds-logs-system.syslog-default-2023.05.17-000007 0 r UNASSIGNED
.ds-logs-endpoint.events.process-default-2023.05.20-000010 0 p STARTED 47276 20.9mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.20-000010 0 r UNASSIGNED
.ds-logs-elastic_agent-default-2023.05.13-000004 0 p STARTED 143 297.8kb 127.0.0.1 xenpher
.ds-logs-elastic_agent-default-2023.05.13-000004 0 r UNASSIGNED
.ds-logs-elastic_agent.fleet_server-default-2023.05.15-000006 0 p STARTED 1001 1.7mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.fleet_server-default-2023.05.15-000006 0 r UNASSIGNED
.ds-logs-system.syslog-default-2023.05.13-000003 0 p STARTED 460835 93.8mb 127.0.0.1 xenpher
.ds-logs-system.syslog-default-2023.05.13-000003 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.20-000010 0 p STARTED 16317 4.1mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.20-000010 0 r UNASSIGNED
.ds-logs-elastic_agent.metricbeat-default-2023.05.13-000004 0 p STARTED 52629 13.3mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.metricbeat-default-2023.05.13-000004 0 r UNASSIGNED
.transform-notifications-000002 0 p STARTED 151 108.4kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.20-000009 0 p STARTED 220 221.2kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.20-000009 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.17-000007 0 p STARTED 449251 97.1mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.17-000007 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.04-000001 0 p STARTED 2226402 457.7mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.04-000001 0 r UNASSIGNED
metrics-endpoint.metadata_current_default 0 p STARTED 11 90.4kb 127.0.0.1 xenpher
.ds-logs-system.auth-default-2023.05.04-000001 0 p STARTED 2807 1.4mb 127.0.0.1 xenpher
.ds-logs-system.auth-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.network-default-2023.05.14-000004 0 p STARTED 60340 15.5mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.network-default-2023.05.14-000004 0 r UNASSIGNED
.ds-logs-endpoint.events.security-default-2023.05.14-000004 0 p STARTED 1212 952.8kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.14-000004 0 r UNASSIGNED
.ds-logs-elastic_agent.metricbeat-default-2023.05.04-000001 0 p STARTED 28152 7mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.metricbeat-default-2023.05.04-000001 0 r UNASSIGNED
.items-default-000001 0 p STARTED 0 225b 127.0.0.1 xenpher
.items-default-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.22-000011 0 p STARTED 43184 11.9mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.22-000011 0 r UNASSIGNED
.ds-metrics-elastic_agent.fleet_server-default-2023.05.04-000001 0 p STARTED 311394 54.2mb 127.0.0.1 xenpher
.ds-metrics-elastic_agent.fleet_server-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-system.system-default-2023.05.13-000003 0 p STARTED 37 235.8kb 127.0.0.1 xenpher
.ds-logs-system.system-default-2023.05.13-000003 0 r UNASSIGNED
.fleet-policies-7 0 p STARTED 47 248.2kb 127.0.0.1 xenpher
.ds-metrics-elastic_agent.metricbeat-default-2023.05.04-000001 0 p STARTED 880034 236.6mb 127.0.0.1 xenpher
.ds-metrics-elastic_agent.metricbeat-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.19-000009 0 p STARTED 60911 12.2mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.19-000009 0 r UNASSIGNED
.ds-logs-system.system-default-2023.05.14-000004 0 p STARTED 393 726.2kb 127.0.0.1 xenpher
.ds-logs-system.system-default-2023.05.14-000004 0 r UNASSIGNED
.ds-logs-system.syslog-default-2023.05.16-000006 0 p STARTED 1075395 190.4mb 127.0.0.1 xenpher
.ds-logs-system.syslog-default-2023.05.16-000006 0 r UNASSIGNED
.ds-logs-elastic_agent.metricbeat-default-2023.05.12-000003 0 p STARTED 52626 13.8mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.metricbeat-default-2023.05.12-000003 0 r UNASSIGNED
.ds-metrics-windows.service-default-2023.05.19-000001 0 p STARTED 1188931 147.9mb 127.0.0.1 xenpher
.ds-metrics-windows.service-default-2023.05.19-000001 0 r UNASSIGNED
.fleet-policies-leader-7 0 p STARTED 3 1.9mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.17-000007 0 p STARTED 49 228.4kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.17-000007 0 r UNASSIGNED
my-data-today-copy-2023.05.12-000001 0 p STARTED 0 247b 127.0.0.1 xenpher
my-data-today-copy-2023.05.12-000001 0 r UNASSIGNED
.fleet-agents-7 0 p STARTED 14 26.3mb 127.0.0.1 xenpher
.ds-logs-system.syslog-default-2023.05.14-000004 0 p STARTED 454633 94mb 127.0.0.1 xenpher
.ds-logs-system.syslog-default-2023.05.14-000004 0 r UNASSIGNED
.ds-logs-elastic_agent.metricbeat-default-2023.05.17-000008 0 p STARTED 91480 22.7mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.metricbeat-default-2023.05.17-000008 0 r UNASSIGNED
.ds-logs-endpoint.events.process-default-2023.05.13-000003 0 p STARTED 11030887 4.2gb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.13-000003 0 r UNASSIGNED
.ds-logs-endpoint.events.library-default-2023.05.12-000002 0 p STARTED 10915 3.4mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.library-default-2023.05.12-000002 0 r UNASSIGNED
.ds-.logs-endpoint.diagnostic.collection-default-2023.05.18-000004 0 p STARTED 29 737.3kb 127.0.0.1 xenpher
.ds-.logs-endpoint.diagnostic.collection-default-2023.05.18-000004 0 r UNASSIGNED
.ds-logs-elastic_agent.fleet_server-default-2023.05.13-000004 0 p STARTED 899 1.6mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.fleet_server-default-2023.05.13-000004 0 r UNASSIGNED
.geoip_databases 0 p STARTED 43 46.8mb 127.0.0.1 xenpher
.ds-logs-endpoint.alerts-default-2023.05.17-000003 0 p STARTED 20 218kb 127.0.0.1 xenpher
.ds-logs-endpoint.alerts-default-2023.05.17-000003 0 r UNASSIGNED
.ds-metrics-system.filesystem-default-2023.05.04-000001 0 p STARTED 352793 83.7mb 127.0.0.1 xenpher
.ds-metrics-system.filesystem-default-2023.05.04-000001 0 r UNASSIGNED
.ds-.logs-deprecation.elasticsearch-default-2023.05.03-000001 0 p STARTED 12 47kb 127.0.0.1 xenpher
.fleet-actions-7 0 p STARTED 10 58.1kb 127.0.0.1 xenpher
.ds-logs-system.security-default-2023.05.13-000003 0 p STARTED 7799 3.6mb 127.0.0.1 xenpher
.ds-logs-system.security-default-2023.05.13-000003 0 r UNASSIGNED
.ds-logs-endpoint.events.network-default-2023.05.15-000005 0 p STARTED 26248 8.2mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.network-default-2023.05.15-000005 0 r UNASSIGNED
.ds-logs-system.syslog-default-2023.05.04-000001 0 p STARTED 4281931 810.8mb 127.0.0.1 xenpher
.ds-logs-system.syslog-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.security-default-2023.05.15-000005 0 p STARTED 832 721.3kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.15-000005 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.18-000008 0 p STARTED 159135 34.6mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.18-000008 0 r UNASSIGNED
.ds-logs-endpoint.alerts-default-2023.05.04-000001 0 p STARTED 682 1.3mb 127.0.0.1 xenpher
.ds-logs-endpoint.alerts-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-system.system-default-2023.05.04-000001 0 p STARTED 9089 3.6mb 127.0.0.1 xenpher
.ds-logs-system.system-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-system.auth-default-2023.05.16-000006 0 p STARTED 576 294.1kb 127.0.0.1 xenpher
.ds-logs-system.auth-default-2023.05.16-000006 0 r UNASSIGNED
.ds-logs-system.application-default-2023.05.16-000006 0 p STARTED 73 213.5kb 127.0.0.1 xenpher
.ds-logs-system.application-default-2023.05.16-000006 0 r UNASSIGNED
.ds-logs-endpoint.events.registry-default-2023.05.16-000006 0 p STARTED 33562 5.8mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.registry-default-2023.05.16-000006 0 r UNASSIGNED
.ds-logs-system.security-default-2023.05.12-000002 0 p STARTED 6968 3mb 127.0.0.1 xenpher
.ds-logs-system.security-default-2023.05.12-000002 0 r UNASSIGNED
.transform-internal-007 0 p STARTED 134 12.2mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.14-000004 0 p STARTED 11068060 4.2gb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.14-000004 0 r UNASSIGNED
```
---
### Author:  [@target\_test](https://discuss.elastic.co/u/target_test)
#### Post date: [May 22, 2023, 7:03am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/2 "2023-05-22T07:03:28Z")
this is the rest of the get command
```auto
.kibana_8.7.1_001 0 p UNASSIGNED
.ds-logs-elastic_agent.filebeat-default-2023.05.16-000007 0 p STARTED 33146 10.3mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.16-000007 0 r UNASSIGNED
.ds-logs-system.system-default-2023.05.16-000006 0 p STARTED 48 265.2kb 127.0.0.1 xenpher
.ds-logs-system.system-default-2023.05.16-000006 0 r UNASSIGNED
.ds-logs-endpoint.events.process-default-2023.05.16-000006 0 p STARTED 10872829 4.1gb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.16-000006 0 r UNASSIGNED
.kibana_task_manager_8.7.1_001 0 p STARTED 754 145.1mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.04-000001 0 p STARTED 23379 5.3mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-system.auth-default-2023.05.14-000004 0 p STARTED 109 301.7kb 127.0.0.1 xenpher
.ds-logs-system.auth-default-2023.05.14-000004 0 r UNASSIGNED
.ds-logs-system.application-default-2023.05.12-000002 0 p STARTED 87 196.1kb 127.0.0.1 xenpher
.ds-logs-system.application-default-2023.05.12-000002 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.22-000012 0 p STARTED 50828 5.7mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.22-000012 0 r UNASSIGNED
.ds-logs-endpoint.events.network-default-2023.05.12-000002 0 p STARTED 48092 14mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.network-default-2023.05.12-000002 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.20-000011 0 p STARTED 3831 643.1kb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.20-000011 0 r UNASSIGNED
.ds-logs-endpoint.events.network-default-2023.05.13-000003 0 p STARTED 46526 13.2mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.network-default-2023.05.13-000003 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.04-000001 0 p STARTED 57469 5.9mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.04-000001 0 r UNASSIGNED
.metrics-endpoint.metadata_united_default 0 p STARTED 14 12.1mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.api-default-2023.05.04-000001 0 p STARTED 82 241.4kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.api-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-elastic_agent.filebeat-default-2023.05.12-000003 0 p STARTED 21409 7.1mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.12-000003 0 r UNASSIGNED
.ds-metrics-system.memory-default-2023.05.04-000001 0 p STARTED 413387 85.1mb 127.0.0.1 xenpher
.ds-metrics-system.memory-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.registry-default-2023.05.12-000002 0 p STARTED 30861 5.6mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.registry-default-2023.05.12-000002 0 r UNASSIGNED
.ds-logs-endpoint.events.security-default-2023.05.16-000006 0 p STARTED 359 385.3kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.16-000006 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.15-000005 0 p STARTED 102063 21.3mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.15-000005 0 r UNASSIGNED
.ds-logs-elastic_agent.metricbeat-default-2023.05.05-000002 0 p STARTED 352375 62mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.metricbeat-default-2023.05.05-000002 0 r UNASSIGNED
.ds-logs-endpoint.events.process-default-2023.05.18-000008 0 p STARTED 2028053 853.5mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.18-000008 0 r UNASSIGNED
.fleet-enrollment-api-keys-7 0 p STARTED 3 19.8kb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.15-000006 0 p STARTED 28123 8mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.15-000006 0 r UNASSIGNED
.ds-logs-system.system-default-2023.05.15-000005 0 p STARTED 299 467.2kb 127.0.0.1 xenpher
.ds-logs-system.system-default-2023.05.15-000005 0 r UNASSIGNED
.ds-logs-elastic_agent.filebeat-default-2023.05.13-000004 0 p STARTED 21421 6.8mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.13-000004 0 r UNASSIGNED
.ds-logs-endpoint.events.network-default-2023.05.17-000007 0 p STARTED 965128 225.3mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.network-default-2023.05.17-000007 0 r UNASSIGNED
.ds-logs-elastic_agent-default-2023.05.05-000002 0 p STARTED 8114 2.9mb 127.0.0.1 xenpher
.ds-logs-elastic_agent-default-2023.05.05-000002 0 r UNASSIGNED
.ds-logs-endpoint.events.library-default-2023.05.14-000004 0 p STARTED 10405 3.5mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.library-default-2023.05.14-000004 0 r UNASSIGNED
.ds-.fleet-actions-results-2023.05.04-000001 0 p STARTED 10 43.6kb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.05-000002 0 p STARTED 129270 29.4mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.filebeat-default-2023.05.05-000002 0 r UNASSIGNED
.ds-logs-endpoint.events.library-default-2023.05.16-000006 0 p STARTED 2387 1.2mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.library-default-2023.05.16-000006 0 r UNASSIGNED
.ds-logs-endpoint.events.process-default-2023.05.15-000005 0 p STARTED 10640012 4gb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.15-000005 0 r UNASSIGNED
.ds-logs-endpoint.events.api-default-2023.05.17-000005 0 p STARTED 4 57.7kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.api-default-2023.05.17-000005 0 r UNASSIGNED
.ds-logs-system.security-default-2023.05.17-000007 0 p STARTED 22244 7.1mb 127.0.0.1 xenpher
.ds-logs-system.security-default-2023.05.17-000007 0 r UNASSIGNED
.ds-logs-endpoint.events.api-default-2023.05.15-000003 0 p STARTED 24 111.8kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.api-default-2023.05.15-000003 0 r UNASSIGNED
.ds-logs-elastic_agent-default-2023.05.17-000008 0 p STARTED 1081 1022kb 127.0.0.1 xenpher
.ds-logs-elastic_agent-default-2023.05.17-000008 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.15-000006 0 p STARTED 51444 5.5mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.15-000006 0 r UNASSIGNED
.ds-logs-endpoint.events.library-default-2023.05.17-000007 0 p STARTED 251402 26.2mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.library-default-2023.05.17-000007 0 r UNASSIGNED
.ds-metrics-system.uptime-default-2023.05.04-000001 0 p STARTED 413443 72.4mb 127.0.0.1 xenpher
.ds-metrics-system.uptime-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-elastic_agent.endpoint_security-default-2023.05.17-000008 0 p STARTED 16328 3.3mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.17-000008 0 r UNASSIGNED
.ds-logs-system.auth-default-2023.05.17-000007 0 p STARTED 510 830.8kb 127.0.0.1 xenpher
.ds-logs-system.auth-default-2023.05.17-000007 0 r UNASSIGNED
.ds-logs-system.security-default-2023.05.14-000004 0 p STARTED 10424 4.9mb 127.0.0.1 xenpher
.ds-logs-system.security-default-2023.05.14-000004 0 r UNASSIGNED
.ds-logs-system.security-default-2023.05.04-000001 0 p STARTED 180412 37.3mb 127.0.0.1 xenpher
.ds-logs-system.security-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-elastic_agent-default-2023.05.16-000007 0 p STARTED 1086 783.4kb 127.0.0.1 xenpher
.ds-logs-elastic_agent-default-2023.05.16-000007 0 r UNASSIGNED
.ds-metrics-elastic_agent.filebeat_input-default-2023.05.04-000001 0 p STARTED 477939 143.9mb 127.0.0.1 xenpher
.ds-metrics-elastic_agent.filebeat_input-default-2023.05.04-000001 0 r UNASSIGNED
.fleet-servers-7 0 p STARTED 3 973.3kb 127.0.0.1 xenpher
.ds-logs-system.syslog-default-2023.05.15-000005 0 p STARTED 450081 88mb 127.0.0.1 xenpher
.ds-logs-system.syslog-default-2023.05.15-000005 0 r UNASSIGNED
.ds-metrics-system.fsstat-default-2023.05.04-000001 0 p STARTED 68942 20.1mb 127.0.0.1 xenpher
.ds-metrics-system.fsstat-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-windows.powershell-default-2023.05.19-000001 0 p STARTED 64 184.3kb 127.0.0.1 xenpher
.ds-logs-windows.powershell-default-2023.05.19-000001 0 r UNASSIGNED
.fleet-artifacts-7 0 p STARTED 15 14.8kb 127.0.0.1 xenpher
.ds-ilm-history-5-2023.05.03-000001 0 p STARTED 1325 340.8kb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.12-000003 0 p STARTED 48049 9.3mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.endpoint_security-default-2023.05.12-000003 0 r UNASSIGNED
.ds-metrics-endpoint.policy-default-2023.05.04-000001 0 p STARTED 107 583kb 127.0.0.1 xenpher
.ds-metrics-endpoint.policy-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-elastic_agent-default-2023.05.15-000006 0 p STARTED 624 524.6kb 127.0.0.1 xenpher
.ds-logs-elastic_agent-default-2023.05.15-000006 0 r UNASSIGNED
.ds-logs-endpoint.events.file-default-2023.05.16-000006 0 p STARTED 107763 21.6mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.file-default-2023.05.16-000006 0 r UNASSIGNED
.ds-metrics-windows.perfmon-default-2023.05.19-000001 0 p STARTED 5092165 333.4mb 127.0.0.1 xenpher
.ds-metrics-windows.perfmon-default-2023.05.19-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.registry-default-2023.05.17-000007 0 p STARTED 1174751 137.1mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.registry-default-2023.05.17-000007 0 r UNASSIGNED
.apm-agent-configuration 0 p STARTED 0 225b 127.0.0.1 xenpher
.ds-logs-system.security-default-2023.05.15-000005 0 p STARTED 5973 2.6mb 127.0.0.1 xenpher
.ds-logs-system.security-default-2023.05.15-000005 0 r UNASSIGNED
.ds-metrics-elastic_agent.elastic_agent-default-2023.05.04-000001 0 p STARTED 1546869 331mb 127.0.0.1 xenpher
.ds-metrics-elastic_agent.elastic_agent-default-2023.05.04-000001 0 r UNASSIGNED
.fleet-files-agent-000001 0 p STARTED 0 225b 127.0.0.1 xenpher
.fleet-file-data-endpoint-000001 0 p STARTED 0 225b 127.0.0.1 xenpher
.ds-logs-elastic_agent.metricbeat-default-2023.05.16-000007 0 p STARTED 75218 15.2mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.metricbeat-default-2023.05.16-000007 0 r UNASSIGNED
.ds-logs-system.auth-default-2023.05.15-000005 0 p STARTED 92 316.5kb 127.0.0.1 xenpher
.ds-logs-system.auth-default-2023.05.15-000005 0 r UNASSIGNED
.security-profile-8 0 p STARTED 1 8.5kb 127.0.0.1 xenpher
my-data-today-2023.05.12-000001 0 p STARTED 0 225b 127.0.0.1 xenpher
my-data-today-2023.05.12-000001 0 r UNASSIGNED
.ds-logs-windows.powershell_operational-default-2023.05.19-000001 0 p STARTED 1 51.6kb 127.0.0.1 xenpher
.ds-logs-windows.powershell_operational-default-2023.05.19-000001 0 r UNASSIGNED
.ds-logs-system.syslog-default-2023.05.12-000002 0 p STARTED 456553 88.6mb 127.0.0.1 xenpher
.ds-logs-system.syslog-default-2023.05.12-000002 0 r UNASSIGNED
.ds-logs-endpoint.events.process-default-2023.05.17-000007 0 p STARTED 2047512 871.4mb 127.0.0.1 xenpher
.ds-logs-endpoint.events.process-default-2023.05.17-000007 0 r UNASSIGNED
.ds-logs-elastic_agent.fleet_server-default-2023.05.04-000001 0 p STARTED 1158 510.9kb 127.0.0.1 xenpher
.ds-logs-elastic_agent.fleet_server-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-endpoint.events.security-default-2023.05.22-000010 0 p STARTED 572 456.2kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.22-000010 0 r UNASSIGNED
.ds-logs-system.application-default-2023.05.14-000004 0 p STARTED 326 491.2kb 127.0.0.1 xenpher
.ds-logs-system.application-default-2023.05.14-000004 0 r UNASSIGNED
.ds-metrics-endpoint.metadata-default-2023.05.04-000001 0 p STARTED 123 322.2kb 127.0.0.1 xenpher
.ds-metrics-endpoint.metadata-default-2023.05.04-000001 0 r UNASSIGNED
.ds-logs-system.system-default-2023.05.17-000007 0 p STARTED 265 605.6kb 127.0.0.1 xenpher
.ds-logs-system.system-default-2023.05.17-000007 0 r UNASSIGNED
.ds-logs-endpoint.events.security-default-2023.05.12-000002 0 p STARTED 459 333.1kb 127.0.0.1 xenpher
.ds-logs-endpoint.events.security-default-2023.05.12-000002 0 r UNASSIGNED
.ds-logs-elastic_agent.metricbeat-default-2023.05.15-000006 0 p STARTED 63584 14.3mb 127.0.0.1 xenpher
.ds-logs-elastic_agent.metricbeat-default-2023.05.15-000006 0 r UNASSIGNED
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 24, 2023, 8:08am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/3 "2023-05-24T08:08:12Z")
What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/cluster-stats.html)?
---
### Author:  [@target\_test](https://discuss.elastic.co/u/target_test)
#### Post date: [May 24, 2023, 8:19am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/4 "2023-05-24T08:19:50Z")
this is the cluster api
```auto
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "elasticsearch",
"cluster_uuid" : "-kMn3M_HS2yUs1AujFSMQg",
"timestamp" : 1684916360111,
"status" : "red",
"indices" : {
"count" : 205,
"shards" : {
"total" : 205,
"primaries" : 205,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"primaries" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
},
"docs" : {
"count" : 144621284,
"deleted" : 1966
},
"store" : {
"size" : "38.8gb",
"size_in_bytes" : 41671121224,
"total_data_set_size" : "38.8gb",
"total_data_set_size_in_bytes" : 41671121224,
"reserved" : "0b",
"reserved_in_bytes" : 0
},
"fielddata" : {
"memory_size" : "992b",
"memory_size_in_bytes" : 992,
"evictions" : 0
},
"query_cache" : {
"memory_size" : "0b",
"memory_size_in_bytes" : 0,
"total_count" : 0,
"hit_count" : 0,
"miss_count" : 0,
"cache_size" : 0,
"cache_count" : 0,
"evictions" : 0
},
"completion" : {
"size" : "0b",
"size_in_bytes" : 0
},
"segments" : {
"count" : 1576,
"memory" : "0b",
"memory_in_bytes" : 0,
"terms_memory" : "0b",
"terms_memory_in_bytes" : 0,
"stored_fields_memory" : "0b",
"stored_fields_memory_in_bytes" : 0,
"term_vectors_memory" : "0b",
"term_vectors_memory_in_bytes" : 0,
"norms_memory" : "0b",
"norms_memory_in_bytes" : 0,
"points_memory" : "0b",
"points_memory_in_bytes" : 0,
"doc_values_memory" : "0b",
"doc_values_memory_in_bytes" : 0,
"index_writer_memory" : "133.7mb",
"index_writer_memory_in_bytes" : 140287904,
"version_map_memory" : "2.1mb",
"version_map_memory_in_bytes" : 2301097,
"fixed_bit_set" : "11.2mb",
"fixed_bit_set_memory_in_bytes" : 11746792,
"max_unsafe_auto_id_timestamp" : 1684915526604,
"file_sizes" : { }
},
"mappings" : {
"total_field_count" : 39788,
"total_deduplicated_field_count" : 17561,
"total_deduplicated_mapping_size" : "122.7kb",
"total_deduplicated_mapping_size_in_bytes" : 125705,
"field_types" : [
{
"name" : "alias",
"count" : 77,
"index_count" : 2,
"script_count" : 0
},
{
"name" : "binary",
"count" : 4,
"index_count" : 4,
"script_count" : 0
},
{
"name" : "boolean",
"count" : 1460,
"index_count" : 179,
"script_count" : 0
},
{
"name" : "byte",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "constant_keyword",
"count" : 672,
"index_count" : 174,
"script_count" : 0
},
{
"name" : "date",
"count" : 937,
"index_count" : 183,
"script_count" : 0
},
{
"name" : "date_nanos",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "date_range",
"count" : 2,
"index_count" : 2,
"script_count" : 0
},
{
"name" : "double",
"count" : 207,
"index_count" : 39,
"script_count" : 0
},
{
"name" : "double_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "flattened",
"count" : 35,
"index_count" : 5,
"script_count" : 0
},
{
"name" : "float",
"count" : 56,
"index_count" : 11,
"script_count" : 0
},
{
"name" : "float_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "geo_point",
"count" : 160,
"index_count" : 80,
"script_count" : 0
},
{
"name" : "geo_shape",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "half_float",
"count" : 3,
"index_count" : 2,
"script_count" : 0
},
{
"name" : "histogram",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "integer",
"count" : 7,
"index_count" : 5,
"script_count" : 0
},
{
"name" : "integer_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "ip",
"count" : 293,
"index_count" : 175,
"script_count" : 0
},
{
"name" : "ip_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "keyword",
"count" : 23855,
"index_count" : 185,
"script_count" : 0
},
{
"name" : "long",
"count" : 2321,
"index_count" : 131,
"script_count" : 0
},
{
"name" : "long_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "match_only_text",
"count" : 781,
"index_count" : 115,
"script_count" : 0
},
{
"name" : "nested",
"count" : 382,
"index_count" : 70,
"script_count" : 0
},
{
"name" : "object",
"count" : 7164,
"index_count" : 182,
"script_count" : 0
},
{
"name" : "scaled_float",
"count" : 43,
"index_count" : 6,
"script_count" : 0
},
{
"name" : "shape",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "short",
"count" : 4,
"index_count" : 2,
"script_count" : 0
},
{
"name" : "text",
"count" : 910,
"index_count" : 177,
"script_count" : 0
},
{
"name" : "unsigned_long",
"count" : 209,
"index_count" : 36,
"script_count" : 0
},
{
"name" : "version",
"count" : 2,
"index_count" : 2,
"script_count" : 0
},
{
"name" : "wildcard",
"count" : 194,
"index_count" : 48,
"script_count" : 0
}
],
"runtime_field_types" : []
},
"analysis" : {
"char_filter_types" : [],
"tokenizer_types" : [],
"filter_types" : [],
"analyzer_types" : [
{
"name" : "pattern",
"count" : 2,
"index_count" : 2
}
],
"built_in_char_filters" : [],
"built_in_tokenizers" : [],
"built_in_filters" : [],
"built_in_analyzers" : [
{
"name" : "powershell_script_analyzer",
"count" : 2,
"index_count" : 2
}
]
},
"versions" : [
{
"version" : "8.7.1",
"index_count" : 206,
"primary_shard_count" : 206,
"total_primary_size" : "38.8gb",
"total_primary_bytes" : 41671121224
}
],
"search" : {
"total" : 121,
"queries" : {
"bool" : 21,
"terms" : 60,
"range" : 10,
"term" : 11
},
"sections" : {
"query" : 71,
"_source" : 10,
"pit" : 1,
"aggs" : 50
}
}
},
"nodes" : {
"count" : {
"total" : 1,
"coordinating_only" : 0,
"data" : 1,
"data_cold" : 1,
"data_content" : 1,
"data_frozen" : 1,
"data_hot" : 1,
"data_warm" : 1,
"index" : 0,
"ingest" : 1,
"master" : 1,
"ml" : 1,
"remote_cluster_client" : 1,
"search" : 0,
"transform" : 1,
"voting_only" : 0
},
"versions" : [
"8.7.1"
],
"os" : {
"available_processors" : 4,
"allocated_processors" : 4,
"names" : [
{
"name" : "Linux",
"count" : 1
}
],
"pretty_names" : [
{
"pretty_name" : "Ubuntu 22.04.2 LTS",
"count" : 1
}
],
"architectures" : [
{
"arch" : "amd64",
"count" : 1
}
],
"mem" : {
"total" : "11.5gb",
"total_in_bytes" : 12431908864,
"adjusted_total" : "11.5gb",
"adjusted_total_in_bytes" : 12431908864,
"free" : "657.8mb",
"free_in_bytes" : 689836032,
"used" : "10.9gb",
"used_in_bytes" : 11742072832,
"free_percent" : 6,
"used_percent" : 94
}
},
"process" : {
"cpu" : {
"percent" : 4
},
"open_file_descriptors" : {
"min" : 1310,
"max" : 1310,
"avg" : 1310
}
},
"jvm" : {
"max_uptime" : "22.5m",
"max_uptime_in_millis" : 1354084,
"versions" : [
{
"version" : "20.0.1",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "20.0.1+9-29",
"vm_vendor" : "Oracle Corporation",
"bundled_jdk" : true,
"using_bundled_jdk" : true,
"count" : 1
}
],
"mem" : {
"heap_used" : "1.9gb",
"heap_used_in_bytes" : 2077008288,
"heap_max" : "6gb",
"heap_max_in_bytes" : 6442450944
},
"threads" : 76
},
"fs" : {
"total" : "227.6gb",
"total_in_bytes" : 244464644096,
"free" : "58.8gb",
"free_in_bytes" : 63212478464,
"available" : "47.2gb",
"available_in_bytes" : 50719744000
},
"plugins" : [],
"network_types" : {
"transport_types" : {
"security4" : 1
},
"http_types" : {
"security4" : 1
}
},
"discovery_types" : {
"single-node" : 1
},
"packaging_types" : [
{
"flavor" : "default",
"type" : "deb",
"count" : 1
}
],
"ingest" : {
"number_of_pipelines" : 79,
"processor_stats" : {
"append" : {
"count" : 160,
"failed" : 0,
"current" : 0,
"time" : "3ms",
"time_in_millis" : 3
},
"community_id" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time" : "0s",
"time_in_millis" : 0
},
"convert" : {
"count" : 1243,
"failed" : 0,
"current" : 0,
"time" : "3ms",
"time_in_millis" : 3
},
"date" : {
"count" : 168202,
"failed" : 0,
"current" : 0,
"time" : "4.2s",
"time_in_millis" : 4281
},
"foreach" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time" : "0s",
"time_in_millis" : 0
},
"geoip" : {
"count" : 10564,
"failed" : 0,
"current" : 0,
"time" : "12s",
"time_in_millis" : 12030
},
"grok" : {
"count" : 472,
"failed" : 6,
"current" : 0,
"time" : "105ms",
"time_in_millis" : 105
},
"gsub" : {
"count" : 412,
"failed" : 0,
"current" : 0,
"time" : "1ms",
"time_in_millis" : 1
},
"kv" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time" : "0s",
"time_in_millis" : 0
},
"pipeline" : {
"count" : 168011,
"failed" : 0,
"current" : 0,
"time" : "364ms",
"time_in_millis" : 364
},
"registered_domain" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time" : "0s",
"time_in_millis" : 0
},
"remove" : {
"count" : 336907,
"failed" : 412,
"current" : 0,
"time" : "493ms",
"time_in_millis" : 493
},
"rename" : {
"count" : 16839,
"failed" : 5871,
"current" : 0,
"time" : "578ms",
"time_in_millis" : 578
},
"script" : {
"count" : 177787,
"failed" : 0,
"current" : 0,
"time" : "1.1s",
"time_in_millis" : 1197
},
"set" : {
"count" : 112532,
"failed" : 0,
"current" : 0,
"time" : "1.2s",
"time_in_millis" : 1215
},
"set_security_user" : {
"count" : 167820,
"failed" : 0,
"current" : 0,
"time" : "1.7s",
"time_in_millis" : 1703
},
"split" : {
"count" : 5,
"failed" : 0,
"current" : 0,
"time" : "0s",
"time_in_millis" : 0
}
}
},
"indexing_pressure" : {
"memory" : {
"current" : {
"combined_coordinating_and_primary" : "0b",
"combined_coordinating_and_primary_in_bytes" : 0,
"coordinating" : "0b",
"coordinating_in_bytes" : 0,
"primary" : "0b",
"primary_in_bytes" : 0,
"replica" : "0b",
"replica_in_bytes" : 0,
"all" : "0b",
"all_in_bytes" : 0
},
"total" : {
"combined_coordinating_and_primary" : "0b",
"combined_coordinating_and_primary_in_bytes" : 0,
"coordinating" : "0b",
"coordinating_in_bytes" : 0,
"primary" : "0b",
"primary_in_bytes" : 0,
"replica" : "0b",
"replica_in_bytes" : 0,
"all" : "0b",
"all_in_bytes" : 0,
"coordinating_rejections" : 0,
"primary_rejections" : 0,
"replica_rejections" : 0
},
"limit" : "0b",
"limit_in_bytes" : 0
}
}
}
}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 24, 2023, 8:21am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/5 "2023-05-24T08:21:56Z")
You will need to share the Elasticsearch logs as well.
---
### Author:  [@target\_test](https://discuss.elastic.co/u/target_test)
#### Post date: [May 24, 2023, 8:26am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/6 "2023-05-24T08:26:38Z")
this is the log using tail -200f
```auto
... 191 more
[2023-05-24T15:24:40,720][WARN][r.suppressed] [xenpher] path: /.kibana_8.7.1_001/_pit, params: {index=.kibana_8.7.1_001, keep_alive=10m}
org.elasticsearch.action.search.SearchPhaseExecutionException:
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:728) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:474) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:217) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1041) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:769) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:390) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:478) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:297) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:77) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:37) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:250) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:606) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:585) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$11(AuthorizationService.java:486) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:989) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:955) ~[?:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:387) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:510) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:111) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1035) ~[?:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:367) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:479) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:416) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:317) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:201) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:299) ~[?:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:216) ~[?:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:191) ~[?:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:147) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:319) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:132) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:203) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$0(RestOpenPointInTimeAction.java:44) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:123) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:113) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:107) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:132) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:440) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:346) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:325) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:202) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:121) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:188) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authc.ServiceAccountAuthenticator.lambda$authenticate$0(ServiceAccountAuthenticator.java:55) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authc.service.ServiceAccountService.lambda$authenticateToken$0(ServiceAccountService.java:136) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) ~[?:?]
at org.elasticsearch.xpack.security.authc.service.CachingServiceAccountTokenStore.lambda$authenticateWithCache$1(CachingServiceAccountTokenStore.java:100) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:510) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:111) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authc.service.CachingServiceAccountTokenStore.authenticateWithCache(CachingServiceAccountTokenStore.java:98) ~[?:?]
at org.elasticsearch.xpack.security.authc.service.CachingServiceAccountTokenStore.authenticate(CachingServiceAccountTokenStore.java:82) ~[?:?]
at org.elasticsearch.xpack.security.authc.service.CompositeServiceAccountTokenStore.lambda$authenticate$0(CompositeServiceAccountTokenStore.java:37) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.service.CachingServiceAccountTokenStore.lambda$authenticateWithCache$2(CachingServiceAccountTokenStore.java:118) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore.doAuthenticate(FileServiceAccountTokenStore.java:82) ~[?:?]
at org.elasticsearch.xpack.security.authc.service.CachingServiceAccountTokenStore.authenticateWithCache(CachingServiceAccountTokenStore.java:110) ~[?:?]
at org.elasticsearch.xpack.security.authc.service.CachingServiceAccountTokenStore.authenticate(CachingServiceAccountTokenStore.java:82) ~[?:?]
at org.elasticsearch.xpack.security.authc.service.CompositeServiceAccountTokenStore.lambda$authenticate$0(CompositeServiceAccountTokenStore.java:37) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
at org.elasticsearch.xpack.security.authc.service.CompositeServiceAccountTokenStore.authenticate(CompositeServiceAccountTokenStore.java:44) ~[?:?]
at org.elasticsearch.xpack.security.authc.service.ServiceAccountService.authenticateToken(ServiceAccountService.java:134) ~[?:?]
at org.elasticsearch.xpack.security.authc.ServiceAccountAuthenticator.authenticate(ServiceAccountAuthenticator.java:53) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:180) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:136) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:95) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:101) ~[?:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:402) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:532) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:317) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:379) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:460) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:353) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.kibana_8.7.1_001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:245) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469) ~[elasticsearch-8.7.1.jar:?]
... 191 more
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 24, 2023, 8:31am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/7 "2023-05-24T08:31:31Z")
We need to see the logs from startup to see what caused the cluster to be red.
---
### Author:  [@target\_test](https://discuss.elastic.co/u/target_test)
#### Post date: [May 24, 2023, 8:34am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/8 "2023-05-24T08:34:12Z")
what command should i use because `cat` show the same output
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 24, 2023, 8:46am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/9 "2023-05-24T08:46:50Z")
Find the logs on the file system and post them here. The location will depend on how you installed Elasticsearch.
---
### Author:  [@target\_test](https://discuss.elastic.co/u/target_test)
#### Post date: [May 24, 2023, 9:01am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/10 "2023-05-24T09:01:39Z")
i copy the log file to my github because the file is too big and when i try to send it here it say server error.
github link
> **[GitHub - PelinCit/log](https://github.com/PelinCit/log)**
>
> Contribute to PelinCit/log development by creating an account on GitHub.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 24, 2023, 9:20am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/11 "2023-05-24T09:20:13Z")
As far as I can see that is only a set of large stack traces from failed searches. There is nothing that indicates what the problem is as far as I can see. Can you restart the node and post the logs from the point where Elasticsearch restarts?
---
### Author:  [@target\_test](https://discuss.elastic.co/u/target_test)
#### Post date: [May 24, 2023, 9:30am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/12 "2023-05-24T09:30:24Z")
this is the log after stopping and starting Elasticsearch
```auto
[2023-05-24T16:23:37,071][INFO][o.e.n.Node] [xenpher] stopped
[2023-05-24T16:23:37,072][INFO][o.e.n.Node] [xenpher] closing ...
[2023-05-24T16:23:37,328][INFO][o.e.i.g.DatabaseReaderLazyLoader] [xenpher] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-9424520871293881299/geoip-databases/Jn2KxqKoTIeiVfCrZZApgw/GeoLite2-Country.mmdb]
[2023-05-24T16:23:37,329][INFO][o.e.i.g.DatabaseReaderLazyLoader] [xenpher] evicted [227] entries from cache after reloading database [/tmp/elasticsearch-9424520871293881299/geoip-databases/Jn2KxqKoTIeiVfCrZZApgw/GeoLite2-ASN.mmdb]
[2023-05-24T16:23:37,330][INFO][o.e.i.g.DatabaseReaderLazyLoader] [xenpher] evicted [227] entries from cache after reloading database [/tmp/elasticsearch-9424520871293881299/geoip-databases/Jn2KxqKoTIeiVfCrZZApgw/GeoLite2-City.mmdb]
[2023-05-24T16:23:37,335][INFO][o.e.n.Node] [xenpher] closed
[2023-05-24T16:24:15,110][INFO][o.e.n.Node] [xenpher] version[8.7.1], pid[997039], build[deb/f229ed3f893a515d590d0f39b05f68913e2d9b53/2023-04-27T04:33:42.127815583Z], OS[Linux/5.19.0-41-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/20.0.1/20.0.1+9-29]
[2023-05-24T16:24:15,142][INFO][o.e.n.Node] [xenpher] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2023-05-24T16:24:15,142][INFO][o.e.n.Node] [xenpher] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms6g, -Xmx6g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-4127784203575546257, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=3221225472, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=deb, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2023-05-24T16:24:26,428][INFO][c.a.c.i.j.JacksonVersion] [xenpher] Package versions: jackson-core=2.13.4, jackson-databind=2.13.4-2, jackson-dataformat-xml=2.13.4, jackson-datatype-jsr310=2.13.4, azure-core=1.34.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
[2023-05-24T16:24:33,944][INFO][o.e.p.PluginsService] [xenpher] loaded module [aggregations]
[2023-05-24T16:24:33,945][INFO][o.e.p.PluginsService] [xenpher] loaded module [analysis-common]
[2023-05-24T16:24:33,945][INFO][o.e.p.PluginsService] [xenpher] loaded module [apm]
[2023-05-24T16:24:33,945][INFO][o.e.p.PluginsService] [xenpher] loaded module [blob-cache]
[2023-05-24T16:24:33,946][INFO][o.e.p.PluginsService] [xenpher] loaded module [constant-keyword]
[2023-05-24T16:24:33,946][INFO][o.e.p.PluginsService] [xenpher] loaded module [data-streams]
[2023-05-24T16:24:33,946][INFO][o.e.p.PluginsService] [xenpher] loaded module [frozen-indices]
[2023-05-24T16:24:33,946][INFO][o.e.p.PluginsService] [xenpher] loaded module [ingest-attachment]
[2023-05-24T16:24:33,947][INFO][o.e.p.PluginsService] [xenpher] loaded module [ingest-common]
[2023-05-24T16:24:33,947][INFO][o.e.p.PluginsService] [xenpher] loaded module [ingest-geoip]
[2023-05-24T16:24:33,947][INFO][o.e.p.PluginsService] [xenpher] loaded module [ingest-user-agent]
[2023-05-24T16:24:33,947][INFO][o.e.p.PluginsService] [xenpher] loaded module [kibana]
[2023-05-24T16:24:33,948][INFO][o.e.p.PluginsService] [xenpher] loaded module [lang-expression]
[2023-05-24T16:24:33,948][INFO][o.e.p.PluginsService] [xenpher] loaded module [lang-mustache]
[2023-05-24T16:24:33,948][INFO][o.e.p.PluginsService] [xenpher] loaded module [lang-painless]
[2023-05-24T16:24:33,949][INFO][o.e.p.PluginsService] [xenpher] loaded module [legacy-geo]
[2023-05-24T16:24:33,949][INFO][o.e.p.PluginsService] [xenpher] loaded module [mapper-extras]
[2023-05-24T16:24:33,949][INFO][o.e.p.PluginsService] [xenpher] loaded module [mapper-version]
[2023-05-24T16:24:33,949][INFO][o.e.p.PluginsService] [xenpher] loaded module [old-lucene-versions]
[2023-05-24T16:24:33,950][INFO][o.e.p.PluginsService] [xenpher] loaded module [parent-join]
[2023-05-24T16:24:33,950][INFO][o.e.p.PluginsService] [xenpher] loaded module [percolator]
[2023-05-24T16:24:33,950][INFO][o.e.p.PluginsService] [xenpher] loaded module [rank-eval]
[2023-05-24T16:24:33,951][INFO][o.e.p.PluginsService] [xenpher] loaded module [reindex]
[2023-05-24T16:24:33,951][INFO][o.e.p.PluginsService] [xenpher] loaded module [repositories-metering-api]
[2023-05-24T16:24:33,951][INFO][o.e.p.PluginsService] [xenpher] loaded module [repository-azure]
[2023-05-24T16:24:33,951][INFO][o.e.p.PluginsService] [xenpher] loaded module [repository-gcs]
[2023-05-24T16:24:33,952][INFO][o.e.p.PluginsService] [xenpher] loaded module [repository-s3]
[2023-05-24T16:24:33,952][INFO][o.e.p.PluginsService] [xenpher] loaded module [repository-url]
[2023-05-24T16:24:33,952][INFO][o.e.p.PluginsService] [xenpher] loaded module [runtime-fields-common]
[2023-05-24T16:24:33,953][INFO][o.e.p.PluginsService] [xenpher] loaded module [search-business-rules]
[2023-05-24T16:24:33,953][INFO][o.e.p.PluginsService] [xenpher] loaded module [searchable-snapshots]
[2023-05-24T16:24:33,953][INFO][o.e.p.PluginsService] [xenpher] loaded module [snapshot-based-recoveries]
[2023-05-24T16:24:33,953][INFO][o.e.p.PluginsService] [xenpher] loaded module [snapshot-repo-test-kit]
[2023-05-24T16:24:33,954][INFO][o.e.p.PluginsService] [xenpher] loaded module [spatial]
[2023-05-24T16:24:33,954][INFO][o.e.p.PluginsService] [xenpher] loaded module [systemd]
[2023-05-24T16:24:33,954][INFO][o.e.p.PluginsService] [xenpher] loaded module [transform]
[2023-05-24T16:24:33,955][INFO][o.e.p.PluginsService] [xenpher] loaded module [transport-netty4]
[2023-05-24T16:24:33,955][INFO][o.e.p.PluginsService] [xenpher] loaded module [unsigned-long]
[2023-05-24T16:24:33,955][INFO][o.e.p.PluginsService] [xenpher] loaded module [vector-tile]
[2023-05-24T16:24:33,955][INFO][o.e.p.PluginsService] [xenpher] loaded module [wildcard]
[2023-05-24T16:24:33,955][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-aggregate-metric]
[2023-05-24T16:24:33,956][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-analytics]
[2023-05-24T16:24:33,956][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-async]
[2023-05-24T16:24:33,956][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-async-search]
[2023-05-24T16:24:33,956][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-autoscaling]
[2023-05-24T16:24:33,957][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-ccr]
[2023-05-24T16:24:33,957][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-core]
[2023-05-24T16:24:33,957][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-deprecation]
[2023-05-24T16:24:33,957][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-enrich]
[2023-05-24T16:24:33,957][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-eql]
[2023-05-24T16:24:33,958][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-fleet]
[2023-05-24T16:24:33,958][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-graph]
[2023-05-24T16:24:33,958][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-identity-provider]
[2023-05-24T16:24:33,958][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-ilm]
[2023-05-24T16:24:33,959][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-logstash]
[2023-05-24T16:24:33,959][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-ml]
[2023-05-24T16:24:33,959][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-monitoring]
[2023-05-24T16:24:33,959][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-profiling]
[2023-05-24T16:24:33,960][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-ql]
[2023-05-24T16:24:33,960][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-rollup]
[2023-05-24T16:24:33,960][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-security]
[2023-05-24T16:24:33,961][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-shutdown]
[2023-05-24T16:24:33,961][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-sql]
[2023-05-24T16:24:33,961][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-stack]
[2023-05-24T16:24:33,961][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-text-structure]
[2023-05-24T16:24:33,962][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-voting-only-node]
[2023-05-24T16:24:33,962][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-watcher]
[2023-05-24T16:24:33,962][INFO][o.e.p.PluginsService] [xenpher] loaded module [x-pack-write-load-forecaster]
[2023-05-24T16:24:33,963][INFO][o.e.p.PluginsService] [xenpher] no plugins loaded
[2023-05-24T16:24:42,145][INFO][o.e.e.NodeEnvironment] [xenpher] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [47.5gb], net total_space [227.6gb], types [ext4]
[2023-05-24T16:24:42,146][INFO][o.e.e.NodeEnvironment] [xenpher] heap size [6gb], compressed ordinary object pointers [true]
[2023-05-24T16:24:47,213][INFO][o.e.n.Node] [xenpher] node name [xenpher], node ID [Jn2KxqKoTIeiVfCrZZApgw], cluster name [elasticsearch], roles [data_cold, data, remote_cluster_client, master, data_warm, data_content, transform, data_hot, ml, data_frozen, ingest]
[2023-05-24T16:24:53,209][INFO][o.e.x.m.p.l.CppLogMessageHandler] [xenpher] [controller/997061] [Main.cc@123] controller (64 bit): Version 8.7.1 (Build e429aa288c3883) Copyright (c) 2023 Elasticsearch BV
[2023-05-24T16:24:55,519][INFO][o.e.x.s.Security] [xenpher] Security is enabled
[2023-05-24T16:24:56,600][INFO][o.e.x.s.a.s.FileRolesStore] [xenpher] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2023-05-24T16:24:57,259][INFO][o.e.x.s.InitialNodeSecurityAutoConfiguration] [xenpher] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2023-05-24T16:24:57,926][INFO][o.e.x.p.ProfilingPlugin] [xenpher] Profiling is enabled
[2023-05-24T16:25:00,123][INFO][o.e.t.n.NettyAllocator] [xenpher] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-05-24T16:25:00,204][INFO][o.e.i.r.RecoverySettings] [xenpher] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-05-24T16:25:00,305][INFO][o.e.d.DiscoveryModule] [xenpher] using discovery type [single-node] and seed hosts providers [settings]
[2023-05-24T16:25:02,626][INFO][o.e.n.Node] [xenpher] initialized
[2023-05-24T16:25:02,627][INFO][o.e.n.Node] [xenpher] starting ...
[2023-05-24T16:25:02,802][INFO][o.e.x.s.c.f.PersistentCache] [xenpher] persistent cache index loaded
[2023-05-24T16:25:02,804][INFO][o.e.x.d.l.DeprecationIndexingComponent] [xenpher] deprecation component started
[2023-05-24T16:25:03,219][INFO][o.e.t.TransportService] [xenpher] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2023-05-24T16:25:07,891][WARN][o.e.c.c.ClusterBootstrapService] [xenpher] this node is locked into cluster UUID [-kMn3M_HS2yUs1AujFSMQg] but [cluster.initial_master_nodes] is set to [xenpher]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.7/important-settings.html#initial_master_nodes
[2023-05-24T16:25:08,157][INFO][o.e.c.s.MasterService] [xenpher] elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {xenpher}{Jn2KxqKoTIeiVfCrZZApgw}{FU2bgVqZSKCSdHZWg1KxIQ}{xenpher}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.1} completing election], term: 16, version: 2923, delta: master node changed {previous [], current [{xenpher}{Jn2KxqKoTIeiVfCrZZApgw}{FU2bgVqZSKCSdHZWg1KxIQ}{xenpher}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.1}]}
[2023-05-24T16:25:08,623][INFO][o.e.c.s.ClusterApplierService] [xenpher] master node changed {previous [], current [{xenpher}{Jn2KxqKoTIeiVfCrZZApgw}{FU2bgVqZSKCSdHZWg1KxIQ}{xenpher}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.1}]}, term: 16, version: 2923, reason: Publication{term=16, version=2923}
[2023-05-24T16:25:08,750][INFO][o.e.r.s.FileSettingsService] [xenpher] starting file settings watcher ...
[2023-05-24T16:25:08,774][INFO][o.e.r.s.FileSettingsService] [xenpher] file settings service up and running [tid=54]
[2023-05-24T16:25:08,781][INFO][o.e.h.AbstractHttpServerTransport] [xenpher] publish_address {192.168.88.240:9200}, bound_addresses {[::]:9200}
[2023-05-24T16:25:08,782][INFO][o.e.n.Node] [xenpher] started {xenpher}{Jn2KxqKoTIeiVfCrZZApgw}{FU2bgVqZSKCSdHZWg1KxIQ}{xenpher}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.1}{xpack.installed=true, ml.allocated_processors_double=4.0, ml.max_jvm_size=6442450944, ml.allocated_processors=4, ml.machine_memory=12431908864}
[2023-05-24T16:25:08,795][INFO][o.e.c.c.NodeJoinExecutor] [xenpher] node-join: [{xenpher}{Jn2KxqKoTIeiVfCrZZApgw}{FU2bgVqZSKCSdHZWg1KxIQ}{xenpher}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.1}] with reason [completing election]
[2023-05-24T16:25:11,492][WARN][o.e.x.s.a.ApiKeyAuthenticator] [xenpher] Authentication using apikey failed - apikey authentication for id Y_uzKIgBgbYXmJZiIrq5 encountered a failure
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.checkGlobalBlock(TransportSingleShardAction.java:111) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.(TransportSingleShardAction.java:145) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:97) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:50) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:250) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:606) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:604) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:600) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:585) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$11(AuthorizationService.java:486) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:989) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:955) ~[?:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:387) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:510) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:111) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1035) ~[?:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:367) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:479) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:416) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:317) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:201) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209) ~[?:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:191) ~[?:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:147) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:319) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:132) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$1(SecurityActionFilter.java:110) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:165) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:121) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:106) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:203) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.get(AbstractClient.java:480) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:216) ~[?:?]
at org.elasticsearch.xpack.security.authc.ApiKeyService.loadApiKeyAndValidateCredentials(ApiKeyService.java:702) ~[?:?]
at org.elasticsearch.xpack.security.authc.ApiKeyService.tryAuthenticate(ApiKeyService.java:658) ~[?:?]
at org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator.authenticate(ApiKeyAuthenticator.java:51) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:180) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158) ~[?:?]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:136) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:95) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:101) ~[?:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:402) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:532) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:317) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:379) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:460) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:353) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
```
i found out after that log the rest showing Authentication using apikey failed for the same id
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 21, 2023, 9:31am UTC](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976/13 "2023-06-21T09:31:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.