# Missing Data in Elasticsearch - Need Help Resolving **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring, elastic-stack-security, elastic-stack-alerting, ilm-index-lifecycle-management\ **Created:** [March 20, 2025, 8:46am UTC](https://discuss.elastic.co/t/missing-data-in-elasticsearch-need-help-resolving/376166 "2025-03-20T08:46:14Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![SalehEska](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saleheska/32/121805_2.png) [@SalehEska](https://discuss.elastic.co/u/SalehEska)\ **Post date:** [March 20, 2025, 8:46am UTC](https://discuss.elastic.co/t/missing-data-in-elasticsearch-need-help-resolving/376166/1 "2025-03-20T08:46:14Z")
Hello Elasticsearch community, I'm currently facing an issue where I have missing data in my Elasticsearch index. I have been investigating the issue, but there have been no errors logged in Logstash itself. The cluster is in a "yellow" status, and there are a large number of unassigned shards. When I try to sync data from my database to Elasticsearch, some records are missing. ### Cluster Health: - **Status** : Yellow - **Number of Nodes** : 1 - **Number of Data Nodes** : 1 - **Active Shards** : 7093 - **Unassigned Shards** : 7058 - **Active Shards Percent** : 50.12% i found this on elastic logs: [2025-03-20T01:38:15,483][WARN][r.suppressed] [AUDITTRAIL-SRV] path: /.kibana\_task\_manager/\_update\_by\_query, params: {ignore\_unavailable=true, refresh=true, index=.kibana\_task\_manager} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:729) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:419) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:761) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:513) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:350) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:620) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1688) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1405) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1541) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1516) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:51) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:37) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:966) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) ~[elasticsearch-8.8.2.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: org.elasticsearch.ElasticsearchException: Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max\_open\_scroll\_context] setting. at org.elasticsearch.search.SearchService.createAndPutReaderContext(SearchService.java:919) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.search.SearchService.createOrGetReaderContext(SearchService.java:897) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:632) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:496) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.8.2.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.8.2.jar:?] ... 3 more ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2025, 8:53am UTC](https://discuss.elastic.co/t/missing-data-in-elasticsearch-need-help-resolving/376166/2 "2025-03-20T08:53:20Z")
> [@SalehEska](#): > > - **Status** : Yellow > - **Number of Nodes** : 1 > - **Number of Data Nodes** : 1 The reason you have yellow status indices is that you have replica shard count set to 1, which can never be allocated on a single-node cluster. > [@SalehEska](#): > > **Active Shards** : 7093 It seems like you have overridden the default shard count per node limits that are there for a reason and now have an exceptionally high number of small shards. I would recommend you carefully read [this section in the official documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html) and reconsider how you are sharding your data. Shards are not free and each add overhead to the cluster. > [@SalehEska](#): > > elasticsearch-8.8.2.jar I would also recommend that you upgrade to the latest version as there has been improvements in how Elasticsearch handle large shard counts in recent versions (am not sure whether the version you are using has these or not). ---
**Author:** ![SalehEska](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saleheska/32/121805_2.png) [@SalehEska](https://discuss.elastic.co/u/SalehEska)\ **Post date:** [March 20, 2025, 9:03am UTC](https://discuss.elastic.co/t/missing-data-in-elasticsearch-need-help-resolving/376166/3 "2025-03-20T09:03:38Z")
Do unassigned shards potentially cause missing data in Elasticsearch? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2025, 9:07am UTC](https://discuss.elastic.co/t/missing-data-in-elasticsearch-need-help-resolving/376166/4 "2025-03-20T09:07:28Z")
Not if they are in yellow status. If they are in red status it does. How have you identified that data is missing? How are you ingesting data? Do you see any errors there? What is the hardware specification of the node? What type of storage are you using? Is it local SSD as recommended [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html#search-use-faster-hardware)? ---
**Author:** ![SalehEska](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saleheska/32/121805_2.png) [@SalehEska](https://discuss.elastic.co/u/SalehEska)\ **Post date:** [March 20, 2025, 9:55am UTC](https://discuss.elastic.co/t/missing-data-in-elasticsearch-need-help-resolving/376166/5 "2025-03-20T09:55:38Z")
I use Logstash to sync data from Oracle to Elasticsearch. I’ve checked the Logstash logs, and there are no exceptions. Logstash syncs the data from Oracle every 30 minutes. To verify the synchronized records, I retrieve the minimum ID from the Oracle table and the maximum synchronized ID from Elasticsearch. I then check the count in Oracle with the following query: ```sql SELECT COUNT(1) FROM table_name WHERE id BETWEEN (minimum_table_id) AND (maximum_synced_id_in_Elasticsearch). ``` Next, I retrieve the count of records with the same IDs from Elasticsearch and compare the two counts. If the counts match, I delete the synchronized records from the Oracle table to prevent significant table growth. Storage type : HDD hardware specs of the node: ``` "os": { "timestamp": 1742464425478, "cpu": { "percent": 11 }, "mem": { "total_in_bytes": 107373125632, "adjusted_total_in_bytes": 107373125632, "free_in_bytes": 27726508032, "used_in_bytes": 79646617600, "free_percent": 26, "used_percent": 74 }, "swap": { "total_in_bytes": 147557699584, "free_in_bytes": 31037906944, "used_in_bytes": 116519792640 } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2025, 10:09am UTC](https://discuss.elastic.co/t/missing-data-in-elasticsearch-need-help-resolving/376166/6 "2025-03-20T10:09:26Z")
> [@SalehEska](#): > > Storage type : HDD As you have a lot of small indices it is possible that you will end up with a lot of small writes when updating/indexing. I would recommend looking at I/O statistics as storage may be a bottleneck. On linux you could run `iostat -x` while indexing is progressing. I am not sure how Logstash reports and handles different types of errors so can't comment on that. It may be wise to increase Logstash logging to check for issues in more detail. > [@SalehEska](#): > > ```auto > "swap": { > "total_in_bytes": 147557699584, > "free_in_bytes": 31037906944, > "used_in_bytes": 116519792640 > } > > ``` When running Elasticsearch it is recommended to have swap completely disabled. I would recommend you address this as swap can cause serious performance issues.