# Search\_phase\_execution\_exception error with all\_shared failes
**URL:** https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169
**Category:** Elasticsearch
**Created:** [May 24, 2023, 5:44am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169 "2023-05-24T05:44:59Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 24, 2023, 5:44am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/1 "2023-05-24T05:44:59Z")
hi team
i am facing this search\_phase\_execution\_exception
Please find the details.
curl -X GET "localhost:9200/\_cluster/health?filter\_path=status,\*\_shards&pretty"
{
"status" : "red",
"active\_primary\_shards" : 0,
"active\_shards" : 0,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 87,
"delayed\_unassigned\_shards" : 0
}
curl -X GET "localhost:9200/\_cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state&pretty"
index shard prirep state node unassigned.reason
casa 4 p UNASSIGNED ALLOCATION\_FAILED
casa 4 r UNASSIGNED CLUSTER\_RECOVERED
casa 1 p UNASSIGNED ALLOCATION\_FAILED
casa 1 r UNASSIGNED CLUSTER\_RECOVERED
casa 3 p UNASSIGNED ALLOCATION\_FAILED
casa 3 r UNASSIGNED CLUSTER\_RECOVERED
casa 2 p UNASSIGNED ALLOCATION\_FAILED
casa 2 r UNASSIGNED CLUSTER\_RECOVERED
casa 0 p UNASSIGNED ALLOCATION\_FAILED
casa 0 r UNASSIGNED CLUSTER\_RECOVERED
datainventory 0 p UNASSIGNED ALLOCATION\_FAILED
datainventory 0 r UNASSIGNED CLUSTER\_RECOVERED
dbcount 0 p UNASSIGNED ALLOCATION\_FAILED
**strong text** dbcount 0 r UNASSIGNED CLUSTER\_RECOVERED
generate\_subnet\_usage 0 p UNASSIGNED ALLOCATION\_FAILED
curl [http://localhost:9200/poddata/poddetails/\_search?size=100](http://localhost:9200/poddata/poddetails/_search?size=100)
{"error":{"root\_cause":,"type":"search\_phase\_execution\_exception","reason":"all shards failed","phase":"query","grouped":true,"failed\_shards":},"status":503}
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 24, 2023, 5:51am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/2 "2023-05-24T05:51:38Z")
It looks like all shards are unassigned. Has something happened to your storage?
Which version of Elasticsearch are you using?
Where is this cluster deployed?
What type of hardware and storage are you using?
What is the size and specification of the cluster?
Can you also please share the Elasticsearch logs.
---
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 24, 2023, 6:11am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/3 "2023-05-24T06:11:39Z")
> [@Christian\_Dahlqvist](#):
>
> Elasticsearch
elasticsearch version
"number" : "6.6.2",
"build\_flavor" : "default",
"build\_type" : "rpm",
"build\_hash" : "3bd3e59",
"build\_date" : "2019-03-06T15:16:26.864148Z",
"build\_snapshot" : false,
"lucene\_version" : "7.6.0",
"minimum\_wire\_compatibility\_version" : "5.6.0",
"minimum\_index\_compatibility\_version" : "5.0.0"
cluster deployed on linux VM
Hard ware 32 GM ram
{
"\_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster\_name" : "elasticsearch",
"cluster\_uuid" : "PnFXlHnwRcKp9fpOo9F-cA",
"timestamp" : 1684908616554,
"status" : "red",
"indices" : {
"count" : 0,
"shards" : { },
"docs" : {
"count" : 0,
"deleted" : 0
},
"store" : {
"size\_in\_bytes" : 0
},
"fielddata" : {
"memory\_size\_in\_bytes" : 0,
"evictions" : 0
},
"query\_cache" : {
"memory\_size\_in\_bytes" : 0,
"total\_count" : 0,
"hit\_count" : 0,
"miss\_count" : 0,
"cache\_size" : 0,
"cache\_count" : 0,
"evictions" : 0
},
"completion" : {
"size\_in\_bytes" : 0
},
"segments" : {
"count" : 0,
"memory\_in\_bytes" : 0,
"terms\_memory\_in\_bytes" : 0,
"stored\_fields\_memory\_in\_bytes" : 0,
"term\_vectors\_memory\_in\_bytes" : 0,
"norms\_memory\_in\_bytes" : 0,
"points\_memory\_in\_bytes" : 0,
"doc\_values\_memory\_in\_bytes" : 0,
"index\_writer\_memory\_in\_bytes" : 0,
"version\_map\_memory\_in\_bytes" : 0,
"fixed\_bit\_set\_memory\_in\_bytes" : 0,
"max\_unsafe\_auto\_id\_timestamp" : -9223372036854775808,
"file\_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 1,
"data" : 1,
"coordinating\_only" : 0,
"master" : 1,
"ingest" : 1
},
"versions" : [
"6.6.2"
],
"os" : {
"available\_processors" : 12,
"allocated\_processors" : 12,
"names" : [
{
"name" : "Linux",
"count" : 1
}
],
"pretty\_names" : [
{
"pretty\_name" : "Linux Server 7.2",
"count" : 1
}
],
"mem" : {
"total\_in\_bytes" : 64427245568,
"free\_in\_bytes" : 50525073408,
"used\_in\_bytes" : 13902172160,
"free\_percent" : 78,
"used\_percent" : 22
}
},
"process" : {
"cpu" : {
"percent" : 0
},
"open\_file\_descriptors" : {
"min" : 325,
"max" : 325,
"avg" : 325
}
},
"jvm" : {
"max\_uptime\_in\_millis" : 71188524,
"versions" : [
{
"version" : "1.8.0\_201",
"vm\_name" : "OpenJDK 64-Bit Server VM",
"vm\_version" : "25.201-b09",
"vm\_vendor" : "Linux",
"count" : 1
}
],
"mem" : {
"heap\_used\_in\_bytes" : 513874816,
"heap\_max\_in\_bytes" : 1037959168
},
"threads" : 68
},
"fs" : {
"total\_in\_bytes" : 375571546112,
"free\_in\_bytes" : 249710772224,
"available\_in\_bytes" : 249710772224
},
"plugins" : ,
"network\_types" : {
"transport\_types" : {
"security4" : 1
},
"http\_types" : {
"security4" : 1
}
}
}
}
i added heap size but istill getting same error
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 24, 2023, 6:20am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/4 "2023-05-24T06:20:08Z")
> [@Kapildev](#):
>
> "number" : "6.6.2",
This is very old and has been EOL for a long time. I would strongly recommend upgrading.
> [@Kapildev](#):
>
> "jvm" : {
> "max\_uptime\_in\_millis" : 71188524,
> "versions" : [
> {
> "version" : "1.8.0\_201",
> "vm\_name" : "OpenJDK 64-Bit Server VM",
> "vm\_version" : "25.201-b09",
> "vm\_vendor" : "Linux",
> "count" : 1
> }
> ],
> "mem" : {
> "heap\_used\_in\_bytes" : 513874816,
> "heap\_max\_in\_bytes" : 1037959168
That is a very small heap size for Elasticsearch. Given the resources available on the host where it is running I would recommend increasing to, e.g. to 2GB.
> [@Christian\_Dahlqvist](#):
>
> Can you also please share the Elasticsearch logs.
Something has happened to your cluster so it is important to share the logs so we can see what Elasticsearch is unhappy about.
---
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 24, 2023, 6:25am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/5 "2023-05-24T06:25:54Z")
> [@Christian\_Dahlqvist](#):
>
> been
[2023-05-24T06:15:53,818][DEBUG][o.e.a.s.TransportSearchAction] [g3GaOxv] All shards failed for phase: [query]
[2023-05-24T06:15:53,818][WARN][r.suppressed] [g3GaOxv] path: /exadatanodedbcount/exadatanodedbcountdetails/\_search, params: {size=1, index=exadatanodedbcount, type=exadatanodedbcountdetails}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_201]
[2023-05-24T06:15:56,824][DEBUG][o.e.a.s.TransportSearchAction] [g3GaOxv] All shards failed for phase: [query]
[2023-05-24T06:15:56,824][WARN][r.suppressed] [g3GaOxv] path: /exadatanodedbcount/exadatanodedbcountdetails/\_search, params: {size=1, index=exadatanodedbcount, type=exadatanodedbcountdetails}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_201]
[2023-05-24T06:16:03,834][DEBUG][o.e.a.s.TransportSearchAction] [g3GaOxv] All shards failed for phase: [query]
[2023-05-24T06:16:03,834][WARN][r.suppressed] [g3GaOxv] path: /exadatanodedbcount/exadatanodedbcountdetails/\_search, params: {size=1, index=exadatanodedbcount, type=exadatanodedbcountdetails}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_201]
i increase the heaop size still getting same erorr
"mem" : {
"total\_in\_bytes" : 64427245568,
"free\_in\_bytes" : 50994049024,
"used\_in\_bytes" : 13433196544,
"free\_percent" : 79,
"used\_percent" : 21
}
},
---
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 24, 2023, 6:36am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/6 "2023-05-24T06:36:05Z")
I increasse the heap size still same issue
"mem" : {
"total\_in\_bytes" : 64427245568,
"free\_in\_bytes" : 50994049024,
"used\_in\_bytes" : 13433196544,
"free\_percent" : 79,
"used\_percent" : 21
}
},
---
### Author:  [@target\_test](https://discuss.elastic.co/u/target_test)
#### Post date: [May 24, 2023, 8:00am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/7 "2023-05-24T08:00:14Z")
sorry for interrupting this but can you help me too because i have a similar problem this is my form [Service unavailable error code 503 all shard failed](https://discuss.elastic.co/t/service-unavailable-error-code-503-all-shard-failed/333976)
---
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 24, 2023, 8:04am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/8 "2023-05-24T08:04:30Z")
```auto
[2023-05-24T08:00:42,619][DEBUG][o.e.a.s.TransportSearchAction] [g3GaOxv] All shards failed for phase: [query]
[2023-05-24T08:00:42,619][WARN][r.suppressed] [g3GaOxv] path: /exadatanodedbcount/exadatanodedbcountdetails/_search, params: {size=1, index=exadatanodedbcount, type=exadatanodedbcountdetails}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
[2023-05-24T08:00:45,627][DEBUG][o.e.a.s.TransportSearchAction] [g3GaOxv] All shards failed for phase: [query]
[2023-05-24T08:00:45,627][WARN][r.suppressed] [g3GaOxv] path: /exadatanodedbcount/exadatanodedbcountdetails/_search, params: {size=1, index=exadatanodedbcount, type=exadatanodedbcountdetails}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
[2023-05-24T08:00:52,639][DEBUG][o.e.a.s.TransportSearchAction] [g3GaOxv] All shards failed for phase: [query]
[2023-05-24T08:00:52,639][WARN][r.suppressed] [g3GaOxv] path: /exadatanodedbcount/exadatanodedbcountdetails/_search, params: {size=1, index=exadatanodedbcount, type=exadatanodedbcountdetails}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
```
Alrady increase JVM memory size
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 24, 2023, 8:09am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/9 "2023-05-24T08:09:04Z")
We will need to see the full log from startup to see why the shards are the way they are.
---
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 24, 2023, 8:46am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/12 "2023-05-24T08:46:53Z")
[2023-05-24T08:22:33,218][WARN][o.e.b.Natives] [unknown] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-2925505177850559673/jna--1985354563/jna8894232920475473689.tmp: /tmp/elasticsearch-2925505177850559673/jna--1985354563/jna8894232920475473689.tmp: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0\_201]
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0\_201]
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0\_201]
at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0\_201]
at java.lang.System.load(System.java:1086) ~[?:1.8.0\_201]
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
at com.sun.jna.Native.(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0\_201]
at java.lang.Class.forName(Class.java:264) ~[?:1.8.0\_201]
at org.elasticsearch.bootstrap.Natives.(Natives.java:45) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:102) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.2.jar:6.6.2]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.2.jar:6.6.2]
[2023-05-24T08:22:33,226][WARN][o.e.b.Natives] [unknown] cannot check if running as root because JNA is not available
[2023-05-24T08:22:33,226][WARN][o.e.b.Natives] [unknown] cannot install system call filter because JNA is not available
[2023-05-24T08:22:33,227][WARN][o.e.b.Natives] [unknown] cannot register console handler because JNA is not available
[2023-05-24T08:22:33,228][WARN][o.e.b.Natives] [unknown] cannot getrlimit RLIMIT\_NPROC because JNA is not available
[2023-05-24T08:22:33,229][WARN][o.e.b.Natives] [unknown] cannot getrlimit RLIMIT\_AS because JNA is not available
[2023-05-24T08:22:33,229][WARN][o.e.b.Natives] [unknown] cannot getrlimit RLIMIT\_FSIZE because JNA is not available
[2023-05-24T08:22:33,512][INFO][o.e.e.NodeEnvironment] [g3GaOxv] using [1] data paths, mounts [[/u01, net usable\_space [232.5gb], net total\_space [349.7gb], types [nfs]
[2023-05-24T08:22:33,513][INFO][o.e.e.NodeEnvironment] [g3GaOxv] heap size [989.8mb], compressed ordinary object pointers [true]
[2023-05-24T08:22:34,579][INFO][o.e.n.Node] [g3GaOxv] node name derived from node ID [g3GaOxvHRgWrWPK5ryD8Ng]; set [node.name] to override
[2023-05-24T08:22:34,579][INFO][o.e.n.Node] [g3GaOxv] version[6.6.2], pid[18066], build[default/rpm/3bd3e59/2019-03-06T15:16:26.864148Z], OS[Linux/4.1.12-124.14.2.el7uek.x86\_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0\_201/25.201-b09]
[2023-05-24T08:22:34,579][INFO][o.e.n.Node] [g3GaOxv] JVM arguments [-Xms2g, -Xmx4g, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2925505177850559673, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs\_err\_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2023-05-24T08:22:36,699][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [aggs-matrix-stats]
[2023-05-24T08:22:36,699][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [analysis-common]
[2023-05-24T08:22:36,699][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [ingest-common]
[2023-05-24T08:22:36,699][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [lang-expression]
[2023-05-24T08:22:36,699][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [lang-mustache]
[2023-05-24T08:22:36,699][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [lang-painless]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [mapper-extras]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [parent-join]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [percolator]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [rank-eval]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [reindex]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [repository-url]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [transport-netty4]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [tribe]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-ccr]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-core]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-deprecation]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-graph]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-ilm]
[2023-05-24T08:22:36,700][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-logstash]
[2023-05-24T08:22:36,701][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-ml]
[2023-05-24T08:22:36,701][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-monitoring]
[2023-05-24T08:22:36,701][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-rollup]
[2023-05-24T08:22:36,701][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-security]
[2023-05-24T08:22:36,701][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-sql]
[2023-05-24T08:22:36,701][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-upgrade]
[2023-05-24T08:22:36,701][INFO][o.e.p.PluginsService] [g3GaOxv] loaded module [x-pack-watcher]
[2023-05-24T08:22:36,701][INFO][o.e.p.PluginsService] [g3GaOxv] no plugins loaded
[2023-05-24T08:22:41,513][INFO][o.e.x.s.a.s.FileRolesStore] [g3GaOxv] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2023-05-24T08:22:42,040][INFO][o.e.x.m.p.l.CppLogMessageHandler] [g3GaOxv] [controller/18171] [Main.cc@109] controller (64 bit): Version 6.6.2 (Build 62531230b275d3) Copyright (c) 2019 Elasticsearch BV
[2023-05-24T08:22:42,644][DEBUG][o.e.a.ActionModule] [g3GaOxv] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2023-05-24T08:22:43,126][INFO][o.e.d.DiscoveryModule] [g3GaOxv] using discovery type [zen] and host providers [settings]
[2023-05-24T08:22:44,126][INFO][o.e.n.Node] [g3GaOxv] initialized
[2023-05-24T08:22:44,126][INFO][o.e.n.Node] [g3GaOxv] starting ...
[2023-05-24T08:22:44,297][INFO][o.e.t.TransportService] [g3GaOxv] publish\_address {127.0.0.1:9300}, bound\_addresses {127.0.0.1:9300}
[2023-05-24T08:22:44,409][WARN][o.e.b.BootstrapChecks] [g3GaOxv] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2023-05-24T08:22:47,465][INFO][o.e.c.s.MasterService] [g3GaOxv] zen-disco-elected-as-master ([0] nodes joined), reason: new\_master {g3GaOxv}{g3GaOxvHRgWrWPK5ryD8Ng}{mTT6OgACRS6kQtGk8nQriw}{localhost}{127.0.0.1:9300}{ml.machine\_memory=64427245568, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}
[2023-05-24T08:22:47,473][INFO][o.e.c.s.ClusterApplierService] [g3GaOxv] new\_master {g3GaOxv}{g3GaOxvHRgWrWPK5ryD8Ng}{mTT6OgACRS6kQtGk8nQriw}{localhost}{127.0.0.1:9300}{ml.machine\_memory=64427245568, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {g3GaOxv}{g3GaOxvHRgWrWPK5ryD8Ng}{mTT6OgACRS6kQtGk8nQriw}{localhost}{127.0.0.1:9300}{ml.machine\_memory=64427245568, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2023-05-24T08:22:47,532][INFO][o.e.h.n.Netty4HttpServerTransport] [g3GaOxv] publish\_address {10.199.6.99:9200}, bound\_addresses {0.0.0.0:9200}
[2023-05-24T08:22:47,534][INFO][o.e.n.Node] [g3GaOxv] started
[2023-05-24T08:22:48,577][INFO][o.e.l.LicenseService] [g3GaOxv] license [a4817da0-aac1-492e-b78c-d8ac6b365b51] mode [basic] - valid
[2023-05-24T08:22:48,592][INFO][o.e.g.GatewayService] [g3GaOxv] recovered [26] indices into cluster\_state
[2023-05-24T08:22:48,946][WARN][o.e.i.e.Engine] [g3GaOxv] [deleteanddeepclean][0] could not lock IndexWriter
org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /u01/elasticsearch/nodes/0/indices/kGZBwqnoRvC7UxChnoYPGQ/0/index/write.lock
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:130) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:105) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:105) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:727) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2199) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2187) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:203) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:168) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1446) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1400) ~[elasticsearch-6.6.2.jar:6.6.2]
---
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 24, 2023, 9:48am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/13 "2023-05-24T09:48:22Z")
Any update on this i added log from scratch
---
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 25, 2023, 5:08am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/14 "2023-05-25T05:08:27Z")
@ Christian\_Dahlqvis
Any update on this issue???
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 25, 2023, 5:52am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/15 "2023-05-25T05:52:45Z")
> [@Kapildev](#):
>
> [2023-05-24T08:22:33,512][INFO][o.e.e.NodeEnvironment] [g3GaOxv] using [1] data paths, mounts [[/u01, net usable\_space [232.5gb], net total\_space [349.7gb], types [nfs]
> [@Kapildev](#):
>
> [2023-05-24T08:22:44,409][WARN][o.e.b.BootstrapChecks] [g3GaOxv] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
> [@Kapildev](#):
>
> [2023-05-24T08:22:48,946][WARN][o.e.i.e.Engine] [g3GaOxv] [deleteanddeepclean][0] could not lock IndexWriter
> org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /u01/elasticsearch/nodes/0/indices/kGZBwqnoRvC7UxChnoYPGQ/0/index/write.lock
What operating system and platform are you running this on?
What type of storage are you using? It does seem like some other process is locking the files Elasticsearch needs. Are you using some form of shared storage, e.g. NFS?
---
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 25, 2023, 5:58am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/16 "2023-05-25T05:58:26Z")
> [@Christian\_Dahlqvist](#):
>
> What type of storage are you using? It does seem like some other process is locking the files Elasticsearch needs. Are you using some form of shared storage, e.g. NFS?
Operating System :-Oracle Linux
We are using sharing storage NFS
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 25, 2023, 6:02am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/17 "2023-05-25T06:02:20Z")
NFS is generally not recommended for Elasticsearch as it can be very slow for an I/O intensive application like Elasticsearch and [has to be mounted correctly for it to work](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/modules-node.html#data-path). I suspect the use of NFS is causing the problems you are seeing. Have a look at the docs I linked to for further details.
---
### Author:  [@Kapildev](https://discuss.elastic.co/u/Kapildev)
#### Post date: [May 25, 2023, 6:18am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/18 "2023-05-25T06:18:32Z")
Yes Actually we have limited resource
so tharts why we are using NFS shared memory.
is there way we can receove he data one time
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 22, 2023, 6:19am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-error-with-all-shared-failes/334169/19 "2023-06-22T06:19:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.