Failed to execute bulk item (update) BulkShardRequest

Elastic Stack Elasticsearch
1

I am facing Live server problem on elastisearch.
My RAM is 32 GB and Heap Memory is 8GB. But still i am facing the problem in elastisearch. Bellow are the logs

// Error Log
[2020-07-26T20:15:05,456][DEBUG][o.e.a.b.TransportShardBulkAction] [ISu_GIR] [vue_storefront_catalog_1_1595322631][0] failed to execute bulk item (update) BulkShardRequest [[vue_storefront_catalog_1_1595322631][0]] containing [146] requests
org.elasticsearch.index.engine.DocumentMissingException: [product][9796]: document missing
at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:93) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:82) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequest(TransportShardBulkAction.java:276) ~[elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:161) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) [elasticsearch-5.6.16.jar:5.6.16]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.jar:5.6.16]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
[2020-07-26T21:07:52,138][INFO ][o.e.c.m.MetaDataCreateIndexService] [ISu_GIR] [t19hfzgnp7-meow] creating index, cause [api], templates , shards [5]/[1], mappings

This issue will breaks the whole functionality of site.
So can any one help me to fix this problem?

2

The last line in the log is worrysome:

[2020-07-26T21:07:52,138][INFO ][o.e.c.m.MetaDataCreateIndexService] [ISu_GIR] [t19hfzgnp7-meow] creating index, cause [api], templates , shards [5]/[1], mappings

See the t19hfzgnp7-meow index? Your cluster has been compromised by the Meow attack, which randomly deletes data from unsecured clusters.

I would urge you to properly secure your cluster very quickly, especially since Elasticsearch Security is FREE as of 6.8.0 and 7.1.0.

2 Likes
3

Thanks @val for your response.
How i can secure the elasticsearch for 5.6.16? Because this the version that i m using.

4

You should consider upgrading to 6.8.0 if that's possible. If not, you really need to at least make sure that your cluster is not publicly accessible anymore.

Also I hope you have snapshots/backups of your data that was deleted.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.