org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (delete-index [logstash-2015.03.21]) within 30s
at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:258)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Is there a way to increase the 30 second time out? Thank you.
Heh. Curator has a --timeout flag, but for some reason I did not include it in the list of flags in the documentation. I'll be fixing that shortly.
$ curator --help
Usage: curator [OPTIONS] COMMAND [ARGS]...
Curator for Elasticsearch indices.
See http://elastic.co/guide/en/elasticsearch/client/curator/current
Options:
--host TEXT Elasticsearch host.
--url_prefix TEXT Elasticsearch http url prefix.
--port INTEGER Elasticsearch port.
--use_ssl Connect to Elasticsearch through SSL.
--http_auth TEXT Use Basic Authentication ex: user:pass
--timeout INTEGER Connection timeout in seconds.
--master-only Only operate on elected master node.
--dry-run Do not perform any changes.
--debug Debug mode
--loglevel TEXT Log level
--logfile TEXT log file
--logformat TEXT Log output format [default|logstash].
--version Show the version and exit.
--help Show this message and exit.
So basically, you only should need to do:
curator --timeout 60 <rest of command-line>
As a matter of advice, if you're getting a 30 second timeout trying to delete an index, that's an indication something is very busy or wrong in your cluster. It shouldn't take more than a few seconds, at most.
curator --debug --timeout 240 --host x.x.x.x delete indices --older-than 100 --time-unit days --timestring '%Y.%m.%d'
It does work for say the first few (each daily index is about 4 gigs), and then tanks after the 30 second timeout. I've already deleted all I need to by just rerunning the above a few times...just thought I'd find a way to do it all in one shot instead of a few times. Thank you.
It also should batch them and delete them all at once (if the list isn't too big, in which case it segments the batch into smaller batches). The fact that it's taking that long to delete them is troubling from a cluster health perspective.
Nevermind. It's not the client timing out. It's Elasticsearch. That's not configurable insofar as I know. As I said, this is indicative of an overtaxed cluster. It shouldn't take this long to delete an index and update the cluster state, hence the error.
How many indices do you have? How many nodes? How many shards per index? You're trying to keep 100 days, so I'm trying to ascertain how big your cluster is, and what's going on.
So, as I understand it, each index is a day, so yes..I am keeping about 100 days. When I run the above command in debug, it would indeed delete about 5 days worth each time. It was almost like the 30 seconds is for the entire operation, not deleting per index. So if I tried to for example just delete one day, it would completely work fine since the entire operation was less then 30 seconds. However if I tried to do say delete 20 days, I'd get about 5 in and then it would fail out...that time took 30 seconds, so again...I suspect this is for an entire operation...which makes sense if it takes 8 seconds to delete a single index, then ya....it would it would only get about 5 until the error. Does that make sense?
Can do...so uh...how do I find out I have one node...but I have no idea how to find out how many shards per index..I have 100 indices (one day = one index). How do I find out how many shards I have per index? Thank you.
logstash-2015.06.17 4 p STARTED 490073 384.3mb 127.0.1.1 Mary Jane Watson
logstash-2015.06.17 4 r UNASSIGNED
logstash-2015.06.17 0 p STARTED 489898 381.1mb 127.0.1.1 Mary Jane Watson
logstash-2015.06.17 0 r UNASSIGNED
logstash-2015.06.17 3 p STARTED 489868 384.5mb 127.0.1.1 Mary Jane Watson
logstash-2015.06.17 3 r UNASSIGNED
logstash-2015.06.17 1 p STARTED 489929 382.6mb 127.0.1.1 Mary Jane Watson
logstash-2015.06.17 1 r UNASSIGNED
logstash-2015.06.17 2 p STARTED 489795 381.5mb 127.0.1.1 Mary Jane Watson
logstash-2015.06.17 2 r UNASSIGNED
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
I have one node...but I have no idea how to find out how many shards per index..I have 100 indices (one day = one index). How do I find out how many shards I have per index? Thank you.