# \[Tool Contribution\] Alfred the ElasticSearch Butler **URL:** \ **Category:** Elasticsearch\ **Created:** [April 2, 2014, 10:57am UTC](https://discuss.elastic.co/t/tool-contribution-alfred-the-elasticsearch-butler/16766 "2014-04-02T10:57:29Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Colton](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@Colton](https://discuss.elastic.co/u/Colton)\ **Post date:** [April 2, 2014, 10:57am UTC](https://discuss.elastic.co/t/tool-contribution-alfred-the-elasticsearch-butler/16766/1 "2014-04-02T10:57:29Z")
Hello ElasticSearch Community, ``` My name is Colton McInroy and I work with DOSarrest Internet ``` Security LTD. Over the past few months I have been working with ElasticSearch fairly closely and building a infrastructure for it. When dealing with lots of indices, managing lots them can be somewhat difficult in most web interfaces we found. We wanted to be able to for instance, have indices over a certain amount of time expire out of the cluster. We came across curator ([https://github.com/elasticsearch/curator](https://github.com/elasticsearch/curator)) which came fairly close, but had some limitations. I decided to spend a couple of days building our own tool from scratch which after discussion we have decided to release to the public via open source. We have called this tool Alfred, after Bruce Wayne's butler Alfred Pennyworth, keeping in line with the Marvel comics theme. ``` Alfred can be set up in a cronjob to automatically groom your ``` indices so that you only keep a certain amount of data, optimize indexes, change settings (such as changing routing), and more. By default no changes are made unless you specify the -r or --run parameter. In its default mode, you can test this tool all you want and get output to see what would have been done without changes actually occurring. You can use the -D option to specify more debug output also if you want to see what's going on (such as "-D debug"). Once you are ready, add the -r parameter and watch Alfred do all the work for you. ``` Alfred was developed in Java, but does not use the ElasticSearch ``` Java API, rather it uses the restful api through the use of Apache HttpClient ([http://hc.apache.org/httpclient-3.x/](http://hc.apache.org/httpclient-3.x/)). The following libraries are included via maven into Alfred... joda-time 2.3 httpcore 4.3.2 gson 2.2.4 httpclient 4.3.3 commons-logging 1.1.3 commons-codec 1.6 commons-cli 1.2 ``` A jar build is located at ``` [https://github.com/DOSarrest-Internet-Security/alfred/raw/master/builds/alfred-0.0.1.jar](https://github.com/DOSarrest-Internet-Security/alfred/raw/master/builds/alfred-0.0.1.jar) Our Github page with source and README is located at > **[DOSarrest-Internet-Security/alfred](https://github.com/DOSarrest-Internet-Security/alfred)** > > alfred - Alfred, the ElasticSearch Butler ``` Here is some of that README file to explain how to use alfred... ``` |usage: alfred -b,--debloom Disable Bloom on Indexes -B,--bloom Enable Bloom on Indexes -c,--close Close Indexes -D,--debug Display debug (debug|info|warn|error|fatal) -d,--delete Delete Indexes -E,--expiresize Byte size limit (Default 10 GB) -e,--expiretime Number of time units old (Default 24) --examples Show some examples of how to use Alfred -f,--flush Flush Indexes -h,--help Help Page (Viewing Now) --host ElasticSearch Host -i,--index Index pattern to match (Default \_all) --max\_num\_segments Optimize max\_num\_segments (Default 2) -o,--optimize Optimize Indexes -O,--open Open Indexes --port ElasticSearch Port -r,--run Required to execute changes on ElasticSearch -s,--style Clean up style (time|size) (Default time) -S,--settings PUT settings --ssl ElasticSearch SSL -T,--time-unit Specify time units (hour|day|none) (Default hour) -t,--timeout ElasticSearch Timeout (Default 30) Alfred Version: 0.0.1| Alfred was built as a tool to handle maintenance work on ElasticSearch. Alfred will delete, flush cache, optimize, close/open, enable/disable bloom filter, as well as put settings on indexes. Alfred can do any of these actions based on either time or size parameters. Examples: |java -jar alfred.jar -e48 -i"cron\_\*" -d | Delete any indexes starting with "cron\_" that are older that 48 hours |java -jar alfred.jar -e24 -i"cron\_\*" -S'{"index.routing.allocation.require.tag":"historical"}' | Set routing to require historical tag on any indexes starting with "cron\_" that are older that 24 hours |java -jar alfred.jar -e24 -i"cron\_\*" -b -o | Disable boom filter and optimize any indexes starting with "cron\_" that are older that 24 hours |java -jar alfred.jar -ssize -E"1 GB" -d | Find all indxes, group by prefix, and delete indexes over a limit of 1 GB. Using the size style with an expire size does not check space based on a single index but rather the indexes adding up over time. Such as the following... |java -jar alfred.jar -i"cron\_\*" -d -ssize -E"500 GB" GENERAL: cron\_2014\_04\_02\_08 is 469.9 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_02\_07 is 436.5 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_02\_06 is 404.0 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_02\_05 is 372.1 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_02\_04 is 341.2 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_02\_03 is 310.1 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_02\_02 is 276.8 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_02\_01 is 240.7 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_02\_00 is 202.2 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_01\_23 is 158.2 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_01\_22 is 110.6 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_01\_21 is 58.6 GiB bytes before the cuttoff. GENERAL: cron\_2014\_04\_01\_20 is 3.1 GiB bytes before the cuttoff. GENERAL: Index cron\_2014\_04\_01\_19 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_18 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_17 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_16 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_15 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_14 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_13 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_12 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_11 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_10 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_09 would have been deleted. GENERAL: Index cron\_2014\_04\_01\_08 would have been deleted. GENERAL: Index cron\_2014\_03\_29\_08 would have been deleted. | If you are using daily indexes, such as the marvel indexes, you could use the following examples to manage them |java -jar alfred.jar -i".marvel-\*" -d -ssize -E"500 GB" | Keep the past 500 GB worth of marvel indices |java -jar alfred.jar -i".marvel-\*" -d -T"day" -e7 | Delete marvel indices older than 7 days old |java -jar alfred.jar -i".marvel-\*" -b -o -T"day" --max\_num\_segments=4 -e1 | Disable bloom filter and optimize marvel indices with max\_num\_segments=4 over 1 day old The following regular expression is used to split indexes into appropriate variables... |^((?[a-zA-Z0-9\.\-_]+)(?(_|-)+)(?[0-9]{4})(?(\.|_|-))(?[0-9]{2})(\.|_|-)(?[0-9]{2})(\.|\_|-)?(?[0-9]{2})?)$ | As long as your indexes following the pattern of this regular expression, Alfred will be glad to manage your indices. ``` The -i parameter is passed to the URL ``` "[http://host](http://host):port/INDEX/\_stats/indices" where "INDEX" is replaced by what ever the -i parameter contains. By default, it does \_all but you can specify all kind of wildcard options. Such as -i".marvel-_", -i"logstash-_", -i"_2014\_04\_02_", etc. Alfred gave us a lot of power to manage our indices, so we thought that the community could use him as well. -- Thanks, Colton McInroy - Director of Security Engineering Phone (Toll Free) _US_ (888)-818-1344 Press 2 _UK_ 0-800-635-0551 Press 2 My Extension 101 24/7 Support [support@dosarrest.com](mailto:support@dosarrest.com) [mailto:support@dosarrest.com](mailto:support@dosarrest.com) Email [colton@dosarrest.com](mailto:colton@dosarrest.com) [mailto:colton@dosarrest.com](mailto:colton@dosarrest.com) Website [http://www.dosarrest.com](http://www.dosarrest.com) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/533BED19.4000608%40dosarrest.com](https://groups.google.com/d/msgid/elasticsearch/533BED19.4000608%40dosarrest.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [April 2, 2014, 5:20pm UTC](https://discuss.elastic.co/t/tool-contribution-alfred-the-elasticsearch-butler/16766/2 "2014-04-02T17:20:45Z")
Colton, Interesting tool and thanks for contributing. Will definitely check it out. One of the main index maintenance tasks that I do is to remove all replicas on older (backup) indices. I am currently doing this task manually because there needs to be human verification of certain criteria before indices are closed/deleted. BTW, Bruce Wayne is DC Comics, not Marvel. Cheers, Ivan (not a comic book reader) On Wed, Apr 2, 2014 at 3:57 AM, Colton [colton@dosarrest.com](mailto:colton@dosarrest.com) wrote: > Hello Elasticsearch Community, > > ``` > My name is Colton McInroy and I work with DOSarrest Internet Security > > ``` > > LTD. Over the past few months I have been working with Elasticsearch fairly > closely and building a infrastructure for it. When dealing with lots of > indices, managing lots them can be somewhat difficult in most web > interfaces we found. We wanted to be able to for instance, have indices > over a certain amount of time expire out of the cluster. We came across > curator ([GitHub - elastic/curator: Curator: Tending your Elasticsearch indices](https://github.com/elasticsearch/curator)) which came fairly > close, but had some limitations. I decided to spend a couple of days > building our own tool from scratch which after discussion we have decided > to release to the public via open source. We have called this tool Alfred, > after Bruce Wayne's butler Alfred Pennyworth, keeping in line with the > Marvel comics theme. > > ``` > Alfred can be set up in a cronjob to automatically groom your indices > > ``` > > so that you only keep a certain amount of data, optimize indexes, change > settings (such as changing routing), and more. By default no changes are > made unless you specify the -r or --run parameter. In its default mode, you > can test this tool all you want and get output to see what would have been > done without changes actually occurring. You can use the -D option to > specify more debug output also if you want to see what's going on (such as > "-D debug"). Once you are ready, add the -r parameter and watch Alfred do > all the work for you. > > ``` > Alfred was developed in Java, but does not use the ElasticSearch Java > > ``` > > API, rather it uses the restful api through the use of Apache HttpClient ( > [HttpClient - HttpClient Home](http://hc.apache.org/httpclient-3.x/)). The following libraries are > included via maven into Alfred... > > joda-time 2.3 > httpcore 4.3.2 > gson 2.2.4 > httpclient 4.3.3 > commons-logging 1.1.3 > commons-codec 1.6 > commons-cli 1.2 > > ``` > A jar build is located at > > ``` > > [https://github.com/DOSarrest-Internet-Security/alfred/raw/master/builds/alfred-0.0.1.jar](https://github.com/DOSarrest-Internet-Security/alfred/raw/master/builds/alfred-0.0.1.jar) > Our Github page with source and README is located at > [GitHub - DOSarrest-Internet-Security/alfred: Alfred, the ElasticSearch Butler](https://github.com/DOSarrest-Internet-Security/alfred) > > ``` > Here is some of that README file to explain how to use alfred... > > ``` > > usage: alfred > -b,--debloom Disable Bloom on Indexes > -B,--bloom Enable Bloom on Indexes > -c,--close Close Indexes > -D,--debug Display debug (debug|info|warn|error|fatal) > -d,--delete Delete Indexes > -E,--expiresize Byte size limit (Default 10 GB) > -e,--expiretime Number of time units old (Default 24) > --examples Show some examples of how to use Alfred > -f,--flush Flush Indexes > -h,--help Help Page (Viewing Now) > --host Elasticsearch Host > -i,--index Index pattern to match (Default \_all) > --max\_num\_segments Optimize max\_num\_segments (Default 2) > -o,--optimize Optimize Indexes > -O,--open Open Indexes > --port Elasticsearch Port > -r,--run Required to execute changes on > Elasticsearch > -s,--style Clean up style (time|size) (Default time) > -S,--settings PUT settings > --ssl Elasticsearch SSL > -T,--time-unit Specify time units (hour|day|none) (Default > hour) > -t,--timeout Elasticsearch Timeout (Default 30) > Alfred Version: 0.0.1 > > Alfred was built as a tool to handle maintenance work on Elasticsearch. > Alfred will delete, flush cache, optimize, close/open, enable/disable bloom > filter, as well as put settings on indexes. Alfred can do any of these > actions based on either time or size parameters. > > Examples: > > java -jar alfred.jar -e48 -i"cron\_\*" -d > > Delete any indexes starting with "cron\_" that are older that 48 hours > > java -jar alfred.jar -e24 -i"cron\_\*" -S'{"index.routing.allocation.require.tag":"historical"}' > > Set routing to require historical tag on any indexes starting with "cron\_" > that are older that 24 hours > > java -jar alfred.jar -e24 -i"cron\_\*" -b -o > > Disable boom filter and optimize any indexes starting with "cron\_" that > are older that 24 hours > > java -jar alfred.jar -ssize -E"1 GB" -d > > Find all indxes, group by prefix, and delete indexes over a limit of 1 GB. > Using the size style with an expire size does not check space based on a > single index but rather the indexes adding up over time. Such as the > following... > > java -jar alfred.jar -i"cron\_\*" -d -ssize -E"500 GB" > GENERAL: cron\_2014\_04\_02\_08 is 469.9 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_02\_07 is 436.5 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_02\_06 is 404.0 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_02\_05 is 372.1 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_02\_04 is 341.2 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_02\_03 is 310.1 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_02\_02 is 276.8 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_02\_01 is 240.7 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_02\_00 is 202.2 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_01\_23 is 158.2 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_01\_22 is 110.6 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_01\_21 is 58.6 GiB bytes before the cuttoff. > GENERAL: cron\_2014\_04\_01\_20 is 3.1 GiB bytes before the cuttoff. > GENERAL: Index cron\_2014\_04\_01\_19 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_18 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_17 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_16 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_15 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_14 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_13 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_12 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_11 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_10 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_09 would have been deleted. > GENERAL: Index cron\_2014\_04\_01\_08 would have been deleted. > GENERAL: Index cron\_2014\_03\_29\_08 would have been deleted. > > If you are using daily indexes, such as the marvel indexes, you could use > the following examples to manage them > > java -jar alfred.jar -i".marvel-\*" -d -ssize -E"500 GB" > > Keep the past 500 GB worth of marvel indices > > java -jar alfred.jar -i".marvel-\*" -d -T"day" -e7 > > Delete marvel indices older than 7 days old > > java -jar alfred.jar -i".marvel-\*" -b -o -T"day" --max\_num\_segments=4 -e1 > > Disable bloom filter and optimize marvel indices with max\_num\_segments=4 > over 1 day old > > The following regular expression is used to split indexes into appropriate > variables... > > ^((?[a-zA-Z0-9\.\-_]+)(?(_|-)+)(?[0-9]{4})(?(\.|_|-))(?[0-9]{2})(\.|_|-)(?[0-9]{2})(\.|\_|-)?(?[0-9]{2})?)$ > > As long as your indexes following the pattern of this regular expression, > Alfred will be glad to manage your indices. > > ``` > The -i parameter is passed to the URL "http://host:port/INDEX/_stats/indices" > > ``` > > where "INDEX" is replaced by what ever the -i parameter contains. By > default, it does \_all but you can specify all kind of wildcard options. > Such as -i".marvel-_", -i"logstash-_", -i"_2014\_04\_02_", etc. Alfred gave > us a lot of power to manage our indices, so we thought that the community > could use him as well. > > -- > Thanks, > Colton McInroy > > - Director of Security Engineering > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/533BED19.4000608%40dosarrest.com](https://groups.google.com/d/msgid/elasticsearch/533BED19.4000608%40dosarrest.com)[https://groups.google.com/d/msgid/elasticsearch/533BED19.4000608%40dosarrest.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/533BED19.4000608%40dosarrest.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDy-wimvtogL4Bg4xLYbeV0gdTNJBvaGTa0YVojqAXNLQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDy-wimvtogL4Bg4xLYbeV0gdTNJBvaGTa0YVojqAXNLQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Colton](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@Colton](https://discuss.elastic.co/u/Colton)\ **Post date:** [April 2, 2014, 7:02pm UTC](https://discuss.elastic.co/t/tool-contribution-alfred-the-elasticsearch-butler/16766/3 "2014-04-02T19:02:15Z")
Hello Ivan, ``` Using the -S option you should be able to easily apply such ``` changes. For example... |java -jar alfred.jar -e24 -i"cron\_\*" -S'{"index":{"number\_of\_replicas":0}}'| ``` This should go through all of the indices started with "cron_" and ``` changing the "number\_of\_replicas" to 0 for any indices over 24 hours. If you are doing this task fairly regularly and have a lot of indices, you may want to make a bash script like the following that runs Alfred. |#!/bin/bash DATE=$(date +"%Y\_%m\_%d" -d "1 days ago") java -jar alfred.jar -e24 -i"cron\_$DATE\*" -S'{"index":{"number\_of\_replicas":0}}'| ``` This would run Alfred so that it goes through all indices starting ``` with "cron\_" that was from yesterday, finding indices old that 24 hours, and changing the number\_of\_replicas... |GENERAL: cron\_2014\_04\_01\_23 is 5 hours above the cuttoff. GENERAL: cron\_2014\_04\_01\_22 is 4 hours above the cuttoff. GENERAL: cron\_2014\_04\_01\_21 is 3 hours above the cuttoff. GENERAL: cron\_2014\_04\_01\_20 is 2 hours above the cuttoff. GENERAL: cron\_2014\_04\_01\_19 is 1 hours above the cuttoff. GENERAL: cron\_2014\_04\_01\_18 is 0 hours above the cuttoff. GENERAL: Index cron\_2014\_04\_01\_17 would have had settings changed for being 1 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_16 would have had settings changed for being 2 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_15 would have had settings changed for being 3 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_14 would have had settings changed for being 4 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_13 would have had settings changed for being 5 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_12 would have had settings changed for being 6 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_11 would have had settings changed for being 7 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_10 would have had settings changed for being 8 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_09 would have had settings changed for being 9 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_08 would have had settings changed for being 10 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_07 would have had settings changed for being 11 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_06 would have had settings changed for being 12 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_05 would have had settings changed for being 13 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_04 would have had settings changed for being 14 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_03 would have had settings changed for being 15 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_02 would have had settings changed for being 16 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_01 would have had settings changed for being 17 hour(s) older than expiry time. GENERAL: Index cron\_2014\_04\_01\_00 would have had settings changed for being 18 hour(s) older than expiry time.| ``` As you may be able to tell, I am not a comic book reader either, so ``` I didn't realize it was DC. Oh well. Thanks, Colton McInroy - Director of Security Engineering Phone (Toll Free) _US_ (888)-818-1344 Press 2 _UK_ 0-800-635-0551 Press 2 My Extension 101 24/7 Support [support@dosarrest.com](mailto:support@dosarrest.com) [mailto:support@dosarrest.com](mailto:support@dosarrest.com) Email [colton@dosarrest.com](mailto:colton@dosarrest.com) [mailto:colton@dosarrest.com](mailto:colton@dosarrest.com) Website [http://www.dosarrest.com](http://www.dosarrest.com) On 4/2/2014 10:20 AM, Ivan Brusic wrote: > Colton, > > Interesting tool and thanks for contributing. Will definitely check it > out. One of the main index maintenance tasks that I do is to remove > all replicas on older (backup) indices. I am currently doing this task > manually because there needs to be human verification of certain > criteria before indices are closed/deleted. > > BTW, Bruce Wayne is DC Comics, not Marvel. > > Cheers, > > Ivan (not a comic book reader) > > On Wed, Apr 2, 2014 at 3:57 AM, Colton \<[colton@dosarrest.com](mailto:colton@dosarrest.com) > [mailto:colton@dosarrest.com](mailto:colton@dosarrest.com)\> wrote: > > ``` > Hello ElasticSearch Community, > > My name is Colton McInroy and I work with DOSarrest Internet > Security LTD. Over the past few months I have been working with > ElasticSearch fairly closely and building a infrastructure for it. > When dealing with lots of indices, managing lots them can be > somewhat difficult in most web interfaces we found. We wanted to > be able to for instance, have indices over a certain amount of > time expire out of the cluster. We came across curator > (https://github.com/elasticsearch/curator) which came fairly > close, but had some limitations. I decided to spend a couple of > days building our own tool from scratch which after discussion we > have decided to release to the public via open source. We have > called this tool Alfred, after Bruce Wayne's butler Alfred > Pennyworth, keeping in line with the Marvel comics theme. > > Alfred can be set up in a cronjob to automatically groom your > indices so that you only keep a certain amount of data, optimize > indexes, change settings (such as changing routing), and more. By > default no changes are made unless you specify the -r or --run > parameter. In its default mode, you can test this tool all you > want and get output to see what would have been done without > changes actually occurring. You can use the -D option to specify > more debug output also if you want to see what's going on (such as > "-D debug"). Once you are ready, add the -r parameter and watch > Alfred do all the work for you. > > Alfred was developed in Java, but does not use the > ElasticSearch Java API, rather it uses the restful api through the > use of Apache HttpClient (http://hc.apache.org/httpclient-3.x/). > The following libraries are included via maven into Alfred... > > joda-time 2.3 > httpcore 4.3.2 > gson 2.2.4 > httpclient 4.3.3 > commons-logging 1.1.3 > commons-codec 1.6 > commons-cli 1.2 > > A jar build is located at > https://github.com/DOSarrest-Internet-Security/alfred/raw/master/builds/alfred-0.0.1.jar > Our Github page with source and README is located at > https://github.com/DOSarrest-Internet-Security/alfred > > Here is some of that README file to explain how to use alfred... > > |usage: alfred > -b,--debloom Disable Bloom on Indexes > -B,--bloom Enable Bloom on Indexes > -c,--close Close Indexes > -D,--debug Display debug (debug|info|warn|error|fatal) > -d,--delete Delete Indexes > -E,--expiresize Byte size limit (Default 10 GB) > -e,--expiretime Number of time units old (Default 24) > --examples Show some examples of how to use Alfred > -f,--flush Flush Indexes > -h,--help Help Page (Viewing Now) > --host ElasticSearch Host > -i,--index Index pattern to match (Default _all) > --max_num_segments Optimize max_num_segments (Default 2) > -o,--optimize Optimize Indexes > -O,--open Open Indexes > --port ElasticSearch Port > -r,--run Required to execute changes on > ElasticSearch > -s,--style Clean up style (time|size) (Default time) > -S,--settings PUT settings > --ssl ElasticSearch SSL > -T,--time-unit Specify time units (hour|day|none) (Default > hour) > -t,--timeout ElasticSearch Timeout (Default 30) > Alfred Version: 0.0.1| > > Alfred was built as a tool to handle maintenance work on > ElasticSearch. Alfred will delete, flush cache, optimize, > close/open, enable/disable bloom filter, as well as put settings > on indexes. Alfred can do any of these actions based on either > time or size parameters. > > Examples: > > |java -jar alfred.jar -e48 -i"cron_*" -d > | > > Delete any indexes starting with "cron_" that are older that 48 hours > > |java -jar alfred.jar -e24 -i"cron_*" -S'{"index.routing.allocation.require.tag":"historical"}' > | > > Set routing to require historical tag on any indexes starting with > "cron_" that are older that 24 hours > > |java -jar alfred.jar -e24 -i"cron_*" -b -o > | > > Disable boom filter and optimize any indexes starting with "cron_" > that are older that 24 hours > > |java -jar alfred.jar -ssize -E"1 GB" -d > | > > Find all indxes, group by prefix, and delete indexes over a limit > of 1 GB. Using the size style with an expire size does not check > space based on a single index but rather the indexes adding up > over time. Such as the following... > > |java -jar alfred.jar -i"cron_*" -d -ssize -E"500 GB" > GENERAL: cron_2014_04_02_08 is 469.9 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_02_07 is 436.5 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_02_06 is 404.0 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_02_05 is 372.1 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_02_04 is 341.2 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_02_03 is 310.1 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_02_02 is 276.8 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_02_01 is 240.7 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_02_00 is 202.2 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_01_23 is 158.2 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_01_22 is 110.6 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_01_21 is 58.6 GiB bytes before the cuttoff. > GENERAL: cron_2014_04_01_20 is 3.1 GiB bytes before the cuttoff. > GENERAL: Index cron_2014_04_01_19 would have been deleted. > GENERAL: Index cron_2014_04_01_18 would have been deleted. > GENERAL: Index cron_2014_04_01_17 would have been deleted. > GENERAL: Index cron_2014_04_01_16 would have been deleted. > GENERAL: Index cron_2014_04_01_15 would have been deleted. > GENERAL: Index cron_2014_04_01_14 would have been deleted. > GENERAL: Index cron_2014_04_01_13 would have been deleted. > GENERAL: Index cron_2014_04_01_12 would have been deleted. > GENERAL: Index cron_2014_04_01_11 would have been deleted. > GENERAL: Index cron_2014_04_01_10 would have been deleted. > GENERAL: Index cron_2014_04_01_09 would have been deleted. > GENERAL: Index cron_2014_04_01_08 would have been deleted. > GENERAL: Index cron_2014_03_29_08 would have been deleted. > | > > If you are using daily indexes, such as the marvel indexes, you > could use the following examples to manage them > > |java -jar alfred.jar -i".marvel-*" -d -ssize -E"500 GB" > | > > Keep the past 500 GB worth of marvel indices > > |java -jar alfred.jar -i".marvel-*" -d -T"day" -e7 > | > > Delete marvel indices older than 7 days old > > |java -jar alfred.jar -i".marvel-*" -b -o -T"day" --max_num_segments=4 -e1 > | > > Disable bloom filter and optimize marvel indices with > max_num_segments=4 over 1 day old > > The following regular expression is used to split indexes into > appropriate variables... > > |^((?[a-zA-Z0-9\\.\\-_]+)(?(_|-)+)(?[0-9]{4})(?(\\.|_|-))(?[0-9]{2})(\\.|_|-)(?[0-9]{2})(\\.|_|-)?(?[0-9]{2})?)$ > | > > As long as your indexes following the pattern of this regular > expression, Alfred will be glad to manage your indices. > > The -i parameter is passed to the URL > "http://host:port/INDEX/_stats/indices" where "INDEX" is replaced > by what ever the -i parameter contains. By default, it does _all > but you can specify all kind of wildcard options. Such as > -i".marvel-*", -i"logstash-*", -i"*2014_04_02*", etc. Alfred gave > us a lot of power to manage our indices, so we thought that the > community could use him as well. > > -- > Thanks, > Colton McInroy > > * Director of Security Engineering > > > Phone > (Toll Free) > _US_ (888)-818-1344 Press 2 > _UK_ 0-800-635-0551 Press 2 > > My Extension 101 > 24/7 Support support@dosarrest.com > Email colton@dosarrest.com > Website http://www.dosarrest.com > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to elasticsearch+unsubscribe@googlegroups.com > . > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/533BED19.4000608%40dosarrest.com > . > For more options, visit https://groups.google.com/d/optout. > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDy-wimvtogL4Bg4xLYbeV0gdTNJBvaGTa0YVojqAXNLQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDy-wimvtogL4Bg4xLYbeV0gdTNJBvaGTa0YVojqAXNLQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDy-wimvtogL4Bg4xLYbeV0gdTNJBvaGTa0YVojqAXNLQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDy-wimvtogL4Bg4xLYbeV0gdTNJBvaGTa0YVojqAXNLQ%40mail.gmail.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/533C5EB7.1020208%40dosarrest.com](https://groups.google.com/d/msgid/elasticsearch/533C5EB7.1020208%40dosarrest.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:38am UTC](https://discuss.elastic.co/t/tool-contribution-alfred-the-elasticsearch-butler/16766/4 "2017-07-06T01:38:40Z")