# Where elasticsearch stores policy? and how it picks up? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 25, 2020, 9:05pm UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622 "2020-07-25T21:05:41Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![abb](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@abb](https://discuss.elastic.co/u/abb)\ **Post date:** [July 25, 2020, 9:05pm UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/1 "2020-07-25T21:05:41Z")
I am using Opendistro for ES 1.9.0 , logstash and kibana. I have created few index templates and policies. Now as per [this](https://stackoverflow.com/questions/27970625/elasticsearch-templates) I placed all my templates. It didn't got picked up.. And also where do I need to copy all my policies and what settings are to be provided. How can I make ES pick index template and policy without APIs. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 26, 2020, 2:13am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/2 "2020-07-26T02:13:08Z")
That is probably a question better asked at the OpenDistro forum. ---
**Author:** ![abb](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@abb](https://discuss.elastic.co/u/abb)\ **Post date:** [July 26, 2020, 4:26am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/3 "2020-07-26T04:26:53Z")
In general opendistro works on top of ES.. Even i didnt find any particular documentation for the same on ES. It only says about on how to create Templates and policies. To link policy to template I have to give policy id in template all by using API, but where to place them and what config has to be provided for policy is not given. I want the templates and policy to be readily available and dont want to do it via API. Can someone help me on this ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 26, 2020, 6:09am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/4 "2020-07-26T06:09:22Z")
Index templates need to be loaded through APIs. They can not be loaded any other way as far as I know. ---
**Author:** ![abb](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@abb](https://discuss.elastic.co/u/abb)\ **Post date:** [July 26, 2020, 7:15am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/5 "2020-07-26T07:15:44Z")
@Christian_Dahlqvist They can also be loaded via logstash, using elasticsearch output plugin. Example ``` output { elasticsearch { template_overwrite => true manage_template => true **template** => "/usr/share/logstash/config/logstash-template.json" template_name => "logstash-template" . . . . . . **ilm_policy => ""** [Not included in my code as of now] } } ``` [Documentation of Output Plugin](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch) I have tried this & it did picked up the template and verified it using GET template/logstash-template, But my question remains the same. How can I load policy? I am aware, that in template I can bind policyid which I have included it in my template and copied it at the same path of the template. ``` "settings" : { "opendistro.index_state_management.policy_id": "delete-logstash-policy", } ``` [Opendistro Documentation of setting](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/settings/) Also Output plugin of ES has a property called **ilm\_policy =\> ""** But documentation of ES is not very clear for **ilm\_policy** & do not tell to provide path of the policy? or just the policyid? Where as it clearly mentions for **template**. Reference: [Template Documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template) [ILM\_POLICY Documentaion](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm_policy) Images: ![Screenshot 2020-07-26 at 12.36.58 PM](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cb4cdd16fbc788756bf75685b13fb971ad274bbb.png) ![Screenshot 2020-07-26 at 12.36.42 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/4/d46c6b8427e82a899565a8fa7a6d4d8d50590027.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 26, 2020, 7:36am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/6 "2020-07-26T07:36:59Z")
I don't think ILM is available in opendistro. ---
**Author:** ![abb](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@abb](https://discuss.elastic.co/u/abb)\ **Post date:** [July 26, 2020, 8:07am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/7 "2020-07-26T08:07:31Z")
Opendistro provides [ISM](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) for ILM. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 26, 2020, 8:23am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/8 "2020-07-26T08:23:12Z")
Logstash can indeed upload templates and do so via the APIs at startup. It does however not support OpenDistros ISM so you will need to set that up separately. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 26, 2020, 9:38am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/9 "2020-07-26T09:38:01Z")
Yeah. As @Christian_Dahlqvist said, it provides another feature which is not ILM or based on ILM. Again, if you really want to use opendistro and not the official distribution, you should really ask in their forum for help. May be they also rewrote a specific plugin for logstash. ---
**Author:** ![abb](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@abb](https://discuss.elastic.co/u/abb)\ **Post date:** [July 26, 2020, 9:58am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/10 "2020-07-26T09:58:04Z")
@Christian_Dahlqvist / @dadoonet Yes, Can you point me to guide / any sample example to do it via logstash in case I opt out of Opendistro?. I am able to load only the template using the logstash output plugin but have no clue on policy loading without using API. ☹ Also, is there a way to apply policies to existing indexes again without using api? (maybe by output plugin) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 26, 2020, 10:07am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/11 "2020-07-26T10:07:37Z")
As far as I know you need to load policies via the API. You can configure it in Kibana, but that uses the API behind the scenes. ---
**Author:** ![abb](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@abb](https://discuss.elastic.co/u/abb)\ **Post date:** [July 26, 2020, 12:24pm UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/12 "2020-07-26T12:24:16Z")
But that will make me to log-in to my multiple env/stage Kibana and then perform api operations manully. ☹ I am not sure if ES provide any feature as such. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 26, 2020, 1:18pm UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/13 "2020-07-26T13:18:07Z")
You could create a script that uploads it through the API. ---
**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\ **Post date:** [July 26, 2020, 4:50pm UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/14 "2020-07-26T16:50:58Z")
I too feel, Elastic should provide generic script to update/automate policies/configItems/dashboards via files. I've written certain scripts, but finds it painful to see which configs have changed using a CI/CD mechanism. What I do is, I compare the "md5sum" of "type" and "visualization" fields of .kibana index to see if the file has changed. It is painful, but wished Elastic Provided an in-house facility ---
**Author:** ![abb](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@abb](https://discuss.elastic.co/u/abb)\ **Post date:** [July 26, 2020, 7:14pm UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/15 "2020-07-26T19:14:21Z")
@Christian_Dahlqvist That was on my list but only as last option.. Coz, i thought I might be missing some links or part of documentation either from ES / Logstash or from Opendistro. @kelk yeah, exactly ES should come up with such features. Could you point me to your scripts, may be I could find something different in approach and also can you explain me more on comparing md5sum, visualization fields of . kibana stuff. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 26, 2020, 7:52pm UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/16 "2020-07-26T19:52:40Z")
If I recall correctly it was possible to load index templates from file in early versions. This caused problems as the templates are applied cluster wide but supplied per node. Files on nodes could overwrite settings that had been changed through APIs. Having the user control this and apply it through APIs based on use case specific rules is IMHO better. ---
**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\ **Post date:** [July 26, 2020, 8:07pm UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/17 "2020-07-26T20:07:00Z")
@abb The idea is , our team puts .kibana dashboard as "files" to make all the environments consistent ( I presume you can extend to policy stores or any configs in ElasticSearch.) So all the files will be in central version control. My script aims to get list of md5sum of the "type" and "visualization" fields (screenshot example) and put it into a csv in every release cycle and put the md5sum of the files too. ![Screenshot 2020-07-26 at 21.04.25](https://us1.discourse-cdn.com/elastic/original/3X/7/0/7050d88f5639db3680434edd2337d0f88c881e3a.png) if anything changes for the file, it will be "POSTED" for those entries. I can't put the script contents out as it is for the company, but wished Elastic provided it as built-in as it is painful to maintain the API list as such as lot of people feels its easy to work on files. ---
**Author:** ![abb](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@abb](https://discuss.elastic.co/u/abb)\ **Post date:** [August 1, 2020, 5:03am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/18 "2020-08-01T05:03:03Z")
So, as there are no other ways to tell ES to pick Policies. I had to create a bash script in the pipeline, which would create policies and index templates one after the other using api(curl) on the master node. Hope ES would come up with features where it would be easy to pass array of policy & template path. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 29, 2020, 5:03am UTC](https://discuss.elastic.co/t/where-elasticsearch-stores-policy-and-how-it-picks-up/242622/19 "2020-08-29T05:03:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.