Hello Experts,
We are running Elastic + Fluentbit + Kibana stack on kubernetes for container logs and it was working correctly with daily rollover based on date(new-YYYY-MM-DD) but on high volume
it caused over shard size issue so created ILM policy mentioned below so that it can rollover quickly. Bootstrapped index is writable but still the old index of (new-YYYY-MM-DD) is getting written instead of the new index new-YYYY-MM-DD-000001. I have mentioned the things tried but no luck yet.
Created new policy with following condition:
PUT /_ilm/policy/new_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age" : "10m" #just to test faster, actually want to set it to an hour.
}
}
},
"delete": {
"min_age": "20d",
"actions": {
"delete": {}
}
}
}
}
}
Created template:
PUT _template/new_template
{
"index_patterns": ["new*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "new_policy",
"index.lifecycle.rollover_alias": "new-alias"
}
}
Bootstrapped new index which is creating and rolling over correctly.
PUT /%3Cnew-%7Bnow%2Fd%7D-000001%3E
{
"aliases": {
"new-alias":{
"is_write_index": true
}
}
}
Output plugin section inside fluentbit-configmap
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
HTTP_User ${FLUENT_ELASTICSEARCH_USER}
HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
Logstash_Format On
Logstash_Prefix new
Trace_Error On
Replace_Dots On
Retry_Limit False
tls On
tls.verify Off
Tried indexing but no luck.
POST _reindex
{
"source": {
"index": "new-2021.09.30"
},
"dest": {
"index": "new-2021.09.30-000001",
"op_type": "create"
}
}
Also, tried creating entire new index-pattern but it seems coming from any system defaults and not making my ILM index as default for new logs.