I have a new installation of elasticsearch, kibana, filebeats, metricbeat, journalbeat and heartbeat. At the time of the new installation it was Version 7.5.0 , now upgraded to V 7.6.0 and today V7.6.1.
I configured the index_templates using:
heartbeat setup -E setup.kibana.username={{ es_user }} -E 'setup.kibana.password={{ es_passwd }}' -E setup.kibana.host={{ kibana_host }}:5601 -E output.logstash.enabled=false -E 'output.elasticsearch.username={{ es_user }}' -E 'output.elasticsearch.password={{ es_passwd }}' -E 'output.elasticsearch.hosts=["{{ elastic_s_host }}:9200"]'
and did not change the index_template afterwards in any way.
Same for
filebeat, metricbeat, journalbeat
Then I Edit index lifecycle policy for heartbeat with the following parameters:
Hot phase: active, Maximum age = 2 min, maximum, Dokuments = 100 , maximum index size 5 megabytes
Warm phase: active, Timing for warm phase : 4 minutes from rollover
for Warm phase I tried both : Move to warm phase on rollover active, and not active.
Cold phase : not active
Delete phase: active, timing for delete phase 6 min from rollover
(I also tried this for all other beats with days instead of minutes without success)
All but the journalbeat indices don't get deleted after the timing given in the delete phase.
instead i get error for all but journalbeat :
illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-7.6.1] does not point to index [filebeat-7.6.1-2020.03.06],
illegal_argument_exception: index.lifecycle.rollover_alias [metricbeat-7.6.1] does not point to index [metricbeat-7.6.1-2020.03.06]
same error also for heartbeat
moreover the indices get roll over to the "warm" phase but get not deleted after the span given in
ILM "Delete phase"
I already google for the error and found some description of the same problem , but no advice i found and tried had helped so far.
i tried also
POST heartbeat-7.6.1/_ilm/retry
results:
{
"acknowledged" : true
}
which gets the above error to disappear, but after some time the error reappears again.
The rollover does not work.
all i get is 2 or more indices:
heartbeat-7.6.1-2020.03.06 containing all the documents the heartbeat has send so far but "Aliases" is empty
several heartbeat-7.6.1-2020.03.06-000001, heartbeat-7.6.1-2020.03.06-000002 indices and so on, which are always empty (document count = 0) and Aliases = heartbeat-7.6.1
but all indices are "open" , current phase is "hot" and current action = rollover
but i never get exactly one hot index containing the latest document and several warm indices containing older documents and deleted indices after the timing given in the delete phase in any index but the journalbeat index
Any help greatly appreciated!
Bodo