I am having trouble finding the path to put my template and policy in ES or Logstash.
I know logstash stores default template path at vendor -> Path
When I referred ES 7.8.0 documentation for policy (Here) & it also says a Note :
Custom ILM policies must already exist on the Elasticsearch cluster before they can be used.
output {
elasticsearch {
ilm_rollover_alias => "custom"
ilm_pattern => "000001"
***ilm_policy => "custom_policy"***
}
}
Does ilm_policy accepts policy path or only policy id? (Check Documentation)
Both documents says policy should already exists on Elastic Cluster.
Can somebody tell me the path where exactly I need to copy my Policy and Template in Elastic Cluster?
or where should I put it in Logstash Cluster config folder?
I have also seen OpenDistro Documentaion (But it talks only about API)
My setup as of now:
I am using Opendistro(1.9.0) for ElasticSearch(7.8.0) along with Logstash(7.8.0) and Kibana on Kubernetes.
I created Index template (logstash-msd-template.json) & Policy (delete-logstash-msd-policy.json) and placed then on Logstash Pod under /config folder
bash-4.2$ pwd
/usr/share/logstash/config
bash-4.2$ ls -ltr
total 26
-rw-r--r-- 1 logstash root 286 Jun 14 21:28 pipelines.yml
-rw-rw-r-- 1 logstash root 342 Jun 14 21:28 logstash-sample.conf
-rw-r--r-- 1 root root 49 Jul 24 13:45 logstash.yml
**-rw-r--r-- 1 root root 1448 Jul 24 13:45 logstash-msd-template.json
-rw-r--r-- 1 root root 982 Jul 24 13:45 delete-logstash-msd-policy.json**
bash-4.2$
and here is my Logstash Output Config and referenced it as below:
output {
elasticsearch {
template_overwrite => true
manage_template => true
template => "/usr/share/logstash/config/logstash-msd-template.json"
template_name => "logstash-msd-template"
**. . .**
ilm_enabled => false
index => "logstash-%{[@metadata][tenant]}-blah-%{+YYYY.MM.dd}"
}
}
My Index Template (logstash-msd-template.json): In which I have referenced policy as ("opendistro.index_state_management.policy_id": "delete-logstash-msd-policy.json")
{
"order": 1,
"version": 60001,
"index_patterns": [
"logstash-msd-*"
],
"settings": {
"opendistro.index_state_management.policy_id": "delete-logstash-msd-policy",
"index": {
"number_of_shards": "1",
"refresh_interval": "5s"
}
},
"mappings": {
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"mapping": {
"norms": false,
"type": "text"
},
"match_mapping_type": "string"
}
},
{
"string_fields": {
"mapping": {
"norms": false,
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
}
}
},
"@version": {
"type": "keyword"
}
}
},
"aliases": {}
}
My Policy (delete-logstash-msd-policy.json):
{
"policy": {
"policy_id": "delete-logstash-msd-policy",
"description": "A simple default policy that changes the states of msd indexes",
"last_updated_time": 1595340380667,
"schema_version": 1,
"error_notification": null,
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "2h"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
]
}
}