Unable to delete indexes - "failed to process cluster event (acquire index lock) within 30s"

I am unable to delete certain indexes in my cluster - using it for logging,
new indexes per day:

curl -XDELETE http://172.16.1.100:9200/2014_03_27
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (acquire index lock) within 30s]","status":503}

curl -XDELETE http://172.16.1.100:9200/2014_04_01
{"ok":true,"acknowledged":true}

curl -XDELETE http://172.16.1.100:9200/2014_03_28
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (acquire index lock) within 30s]","status":503}

curl -XDELETE http://172.16.1.100:9200/2014_04_02
{"ok":true,"acknowledged":true}

curl -XDELETE http://172.16.1.100:9200/2014_04_03
{"ok":true,"acknowledged":true}

However, I could deleted my 2014_04_01 and _02/_03 index just fine. Here
are logs (showing two errors above and successful delete of 2014_04_01/2/3):

[2014-04-09 13:56:56,323][DEBUG][action.admin.indices.delete] [bAggregator00
] [2014_03_27] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:failed to process cluster
event (acquire index lock) within 30s
at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService$1.
run(MetaDataDeleteIndexService.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-04-09 13:59:21,140][INFO ][cluster.metadata ] [bAggregator00]
[2014_04_01] deleting index
[2014-04-09 14:00:29,773][DEBUG][action.admin.indices.delete] [bAggregator00
] [2014_03_28] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:failed to process cluster
event (acquire index lock) within 30s
at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService$1.
run(MetaDataDeleteIndexService.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-04-09 14:03:29,350][INFO ][cluster.metadata ] [bAggregator00]
[2014_04_02] deleting index
[2014-04-09 14:05:03,164][INFO ][cluster.metadata ] [bAggregator00]
[2014_04_03] deleting index

Any ideas/help would be appreciated, thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fe3471ea-34e6-4cd1-b8d4-042607555eda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Similar error

[2014-04-16 16:57:08,464][DEBUG][action.admin.indices.create] [Jerry Jaxon]
[myindex-2014.04.16] failed to create
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (acquire index lock) within 30s
at
org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:141)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

On Wednesday, April 9, 2014 3:10:25 PM UTC+1, Zachary Lammers wrote:

I am unable to delete certain indexes in my cluster - using it for
logging, new indexes per day:

curl -XDELETE http://172.16.1.100:9200/2014_03_27
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (acquire index lock) within 30s]","status":503}

curl -XDELETE http://172.16.1.100:9200/2014_04_01
{"ok":true,"acknowledged":true}

curl -XDELETE http://172.16.1.100:9200/2014_03_28
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (acquire index lock) within 30s]","status":503}

curl -XDELETE http://172.16.1.100:9200/2014_04_02
{"ok":true,"acknowledged":true}

curl -XDELETE http://172.16.1.100:9200/2014_04_03
{"ok":true,"acknowledged":true}

However, I could deleted my 2014_04_01 and _02/_03 index just fine. Here
are logs (showing two errors above and successful delete of 2014_04_01/2/3):

[2014-04-09 13:56:56,323][DEBUG][action.admin.indices.delete] [
bAggregator00] [2014_03_27] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:failed to process cluster
event (acquire index lock) within 30s
at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService$1
.run(MetaDataDeleteIndexService.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-04-09 13:59:21,140][INFO ][cluster.metadata ] [bAggregator00
] [2014_04_01] deleting index
[2014-04-09 14:00:29,773][DEBUG][action.admin.indices.delete] [
bAggregator00] [2014_03_28] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:failed to process cluster
event (acquire index lock) within 30s
at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService$1
.run(MetaDataDeleteIndexService.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-04-09 14:03:29,350][INFO ][cluster.metadata ] [bAggregator00
] [2014_04_02] deleting index
[2014-04-09 14:05:03,164][INFO ][cluster.metadata ] [bAggregator00
] [2014_04_03] deleting index

Any ideas/help would be appreciated, thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a21a8618-2ec5-40c3-a410-5a301c876ff0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

I'm also seeing the same issue as described. Does anyone know what may be
causing this?

Would appreciate your assistance.

Rafi

On Wednesday, April 16, 2014 7:58:39 PM UTC+3, nicktgr15 wrote:

Similar error

[2014-04-16 16:57:08,464][DEBUG][action.admin.indices.create] [Jerry
Jaxon] [myindex-2014.04.16] failed to create
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (acquire index lock) within 30s
at
org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:141)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

On Wednesday, April 9, 2014 3:10:25 PM UTC+1, Zachary Lammers wrote:

I am unable to delete certain indexes in my cluster - using it for
logging, new indexes per day:

curl -XDELETE http://172.16.1.100:9200/2014_03_27
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (acquire index lock) within 30s]","status":503}

curl -XDELETE http://172.16.1.100:9200/2014_04_01
{"ok":true,"acknowledged":true}

curl -XDELETE http://172.16.1.100:9200/2014_03_28
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (acquire index lock) within 30s]","status":503}

curl -XDELETE http://172.16.1.100:9200/2014_04_02
{"ok":true,"acknowledged":true}

curl -XDELETE http://172.16.1.100:9200/2014_04_03
{"ok":true,"acknowledged":true}

However, I could deleted my 2014_04_01 and _02/_03 index just fine. Here
are logs (showing two errors above and successful delete of 2014_04_01/2/3):

[2014-04-09 13:56:56,323][DEBUG][action.admin.indices.delete] [
bAggregator00] [2014_03_27] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (acquire index lock) within 30s
at org.elasticsearch.cluster.metadata.
MetaDataDeleteIndexService$1.run(MetaDataDeleteIndexService.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-04-09 13:59:21,140][INFO ][cluster.metadata ] [
bAggregator00] [2014_04_01] deleting index
[2014-04-09 14:00:29,773][DEBUG][action.admin.indices.delete] [
bAggregator00] [2014_03_28] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (acquire index lock) within 30s
at org.elasticsearch.cluster.metadata.
MetaDataDeleteIndexService$1.run(MetaDataDeleteIndexService.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-04-09 14:03:29,350][INFO ][cluster.metadata ] [
bAggregator00] [2014_04_02] deleting index
[2014-04-09 14:05:03,164][INFO ][cluster.metadata ] [
bAggregator00] [2014_04_03] deleting index

Any ideas/help would be appreciated, thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/944353d4-f9af-44f2-8ed2-20393e2c328f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

I am facing similar exceptions with elasticsearch 0.90. Wondering if there
was any follow up on this in the community.

thanks,
Nikhil

On Wednesday, April 16, 2014 9:58:39 AM UTC-7, nicktgr15 wrote:

Similar error

[2014-04-16 16:57:08,464][DEBUG][action.admin.indices.create] [Jerry
Jaxon] [myindex-2014.04.16] failed to create
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (acquire index lock) within 30s
at
org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:141)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

On Wednesday, April 9, 2014 3:10:25 PM UTC+1, Zachary Lammers wrote:

I am unable to delete certain indexes in my cluster - using it for
logging, new indexes per day:

curl -XDELETE http://172.16.1.100:9200/2014_03_27
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (acquire index lock) within 30s]","status":503}

curl -XDELETE http://172.16.1.100:9200/2014_04_01
{"ok":true,"acknowledged":true}

curl -XDELETE http://172.16.1.100:9200/2014_03_28
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (acquire index lock) within 30s]","status":503}

curl -XDELETE http://172.16.1.100:9200/2014_04_02
{"ok":true,"acknowledged":true}

curl -XDELETE http://172.16.1.100:9200/2014_04_03
{"ok":true,"acknowledged":true}

However, I could deleted my 2014_04_01 and _02/_03 index just fine. Here
are logs (showing two errors above and successful delete of 2014_04_01/2/3):

[2014-04-09 13:56:56,323][DEBUG][action.admin.indices.delete] [
bAggregator00] [2014_03_27] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (acquire index lock) within 30s
at org.elasticsearch.cluster.metadata.
MetaDataDeleteIndexService$1.run(MetaDataDeleteIndexService.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-04-09 13:59:21,140][INFO ][cluster.metadata ] [
bAggregator00] [2014_04_01] deleting index
[2014-04-09 14:00:29,773][DEBUG][action.admin.indices.delete] [
bAggregator00] [2014_03_28] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (acquire index lock) within 30s
at org.elasticsearch.cluster.metadata.
MetaDataDeleteIndexService$1.run(MetaDataDeleteIndexService.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-04-09 14:03:29,350][INFO ][cluster.metadata ] [
bAggregator00] [2014_04_02] deleting index
[2014-04-09 14:05:03,164][INFO ][cluster.metadata ] [
bAggregator00] [2014_04_03] deleting index

Any ideas/help would be appreciated, thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cb3e291f-8646-4c26-927d-26242b930d1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I am getting the same issue . Is there a way to force delete the index through API or from the disk?

Any suggestions would be really helpful.

We found the issue. Just restart the master node and force the cluster to choose different master.

In my case the issue was something went wrong while acquiring lock from master. So I changed the master. It worked :slight_smile:

I met the issus too ,but how to solve the probem .when my machine met the problem ,the systemctl is disabled and timeout