Objective:
Write a Logstash configuration that receives logs and sends it to an index using an ILM policy, and rolls over an index after a certain threshold (max 10 documents).
Below is my Logstash configuration for the output plugin.
Logstash:
output {
elasticsearch {
hosts => ["${HOST}"]
user => "${USERNAME}"
password => "${PASSWORD}"
index => "cloudwatch-testing"
template_name => "cloudwatch"
ilm_rollover_alias => "cloudwatch-testing-alias"
ilm_pattern => "000001"
ilm_policy => "cloudwatch-policy"
id => "cloudwatch"
}
stdout { codec => rubydebug }
}
At first, I assumed this will do the work for me but then I realized that if cloudwatch-testing-000001 doesn't exist, then it will write the logs to cloudwatch-testing-alias instead. Therefore, I found these Elasticsearch queries that can create a template and then roll over an index such as cloudwatch-testing-000001 to cloudwatch-testing-000002.
Problem: This process is manual. I need an automated solution where I can just send my logs, and after every 10 logs, the index is rolled over and a new index is created. What suggestions / feedback do you have in mind?
Elasticsearch queries:
1- Create a template using an existing policy
PUT _template/cloudwatch
{
"index_patterns": ["cloudwatch-testing*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "cloudwatch-policy",
"index.lifecycle.rollover_alias": "cloudwatch-testing-alias"
}
}
2- Create a new index for logstash to write to, if not it will write to cloudwatch-testing-alias instead which is not the index I would like to write to.
PUT cloudwatch-testing-000001
{
"aliases": {
"cloudwatch-testing-alias":{
"is_write_index": true,
"rolled_over" : true
}
}
}
3- The below query command rolls the index over manually, but I would like to do this step automatically.
POST /cloudwatch-testing-alias/_rollover/cloudwatch-testing-000002
{
"conditions": {
"max_docs": 10
},
"settings": {
"index.number_of_shards": 1
}
}
Policy Attributes:
GET /_ilm/policy/cloudwatch-policy
Returns:
{
"cloudwatch-policy" : {
"version" : 3,
"modified_date" : "2020-01-09T23:29:44.942Z",
"policy" : {
"phases" : {
"warm" : {
"min_age" : "30d",
"actions" : {
"set_priority" : {
"priority" : 50
}
}
},
"cold" : {
"min_age" : "30d",
"actions" : {
"freeze" : { },
"set_priority" : {
"priority" : 0
}
}
},
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "30d",
"max_docs" : 10
},
"set_priority" : {
"priority" : 100
}
}
},
"delete" : {
"min_age" : "180d",
"actions" : {
"delete" : { }
}
}
}
}
}
}