Where elasticsearch stores policy? and how it picks up?

[abb]

I am using Opendistro for ES 1.9.0 , logstash and kibana.

I have created few index templates and policies.

Now as per this I placed all my templates.

It didn't got picked up.. And also where do I need to copy all my policies and what settings are to be provided.

How can I make ES pick index template and policy without APIs.

[Christian_Dahlqvist]

That is probably a question better asked at the OpenDistro forum.

[abb]

In general opendistro works on top of ES.. Even i didnt find any particular documentation for the same on ES.

It only says about on how to create Templates and policies. To link policy to template I have to give policy id in template all by using API, but where to place them and what config has to be provided for policy is not given.

I want the templates and policy to be readily available and dont want to do it via API.

Can someone help me on this

[Christian_Dahlqvist]

Index templates need to be loaded through APIs. They can not be loaded any other way as far as I know.

[abb]

@Christian_Dahlqvist They can also be loaded via logstash, using elasticsearch output plugin.

Example

output { elasticsearch {
        template_overwrite => true
        manage_template => true
        **template** => "/usr/share/logstash/config/logstash-template.json"
        template_name => "logstash-template"
        . . . . . . 
       **ilm_policy => "<policy>"** [Not included in my code as of now]
     }
  }

Documentation of Output Plugin

I have tried this & it did picked up the template and verified it using GET template/logstash-template,

But my question remains the same. How can I load policy?

I am aware, that in template I can bind policyid which I have included it in my template and copied it at the same path of the template.

"settings" : {
    "opendistro.index_state_management.policy_id": "delete-logstash-policy",
}

Opendistro Documentation of setting

Also Output plugin of ES has a property called ilm_policy => ""

But documentation of ES is not very clear for ilm_policy & do not tell to provide path of the policy? or just the policyid? Where as it clearly mentions for template.

Reference:
Template Documentation
ILM_POLICY Documentaion

Images:

[dadoonet (David Pilato)]

I don't think ILM is available in opendistro.

[abb]

Opendistro provides ISM for ILM.

[Christian_Dahlqvist]

Logstash can indeed upload templates and do so via the APIs at startup. It does however not support OpenDistros ISM so you will need to set that up separately.

[dadoonet (David Pilato)]

Yeah. As @Christian_Dahlqvist said, it provides another feature which is not ILM or based on ILM.

Again, if you really want to use opendistro and not the official distribution, you should really ask in their forum for help. May be they also rewrote a specific plugin for logstash.

[abb]

@Christian_Dahlqvist / @dadoonet Yes, Can you point me to guide / any sample example to do it via logstash in case I opt out of Opendistro?. I am able to load only the template using the logstash output plugin but have no clue on policy loading without using API.

Also, is there a way to apply policies to existing indexes again without using api? (maybe by output plugin)

[Christian_Dahlqvist]

As far as I know you need to load policies via the API. You can configure it in Kibana, but that uses the API behind the scenes.

[abb]

But that will make me to log-in to my multiple env/stage Kibana and then perform api operations manully. I am not sure if ES provide any feature as such.

[Christian_Dahlqvist]

You could create a script that uploads it through the API.

[kelk (kin)]

I too feel, Elastic should provide generic script to update/automate policies/configItems/dashboards via files.
I've written certain scripts, but finds it painful to see which configs have changed using a CI/CD mechanism.

What I do is, I compare the "md5sum" of "type" and "visualization" fields of .kibana index to see if the file has changed. It is painful, but wished Elastic Provided an in-house facility

[abb]

@Christian_Dahlqvist That was on my list but only as last option.. Coz, i thought I might be missing some links or part of documentation either from ES / Logstash or from Opendistro.

@kelk yeah, exactly ES should come up with such features.

Could you point me to your scripts, may be I could find something different in approach and also can you explain me more on comparing md5sum, visualization fields of . kibana stuff.

[Christian_Dahlqvist]

If I recall correctly it was possible to load index templates from file in early versions. This caused problems as the templates are applied cluster wide but supplied per node. Files on nodes could overwrite settings that had been changed through APIs. Having the user control this and apply it through APIs based on use case specific rules is IMHO better.

[kelk (kin)]

@abb
The idea is , our team puts .kibana dashboard as "files" to make all the environments consistent ( I presume you can extend to policy stores or any configs in ElasticSearch.) So all the files will be in central version control.
My script aims to get list of md5sum of the "type" and "visualization" fields (screenshot example) and put it into a csv in every release cycle and put the md5sum of the files too.

if anything changes for the file, it will be "POSTED" for those entries.

I can't put the script contents out as it is for the company, but wished Elastic provided it as built-in as it is painful to maintain the API list as such as lot of people feels its easy to work on files.

[abb]

So, as there are no other ways to tell ES to pick Policies.

I had to create a bash script in the pipeline, which would create policies and index templates one after the other using api(curl) on the master node.

Hope ES would come up with features where it would be easy to pass array of policy & template path.