ILM is not working as per parameter

HI,

here is my policy and template and attached conf file, 4100+ documents ingested successfully but it is not rolled over on 1000 doc check please help me out why it is not rolling over

Here is my conf content

input {
file {
codec => "json"
path => "/home/shoaib/searlite_latest_data.json"
start_position => "beginning"
sincedb_path => "/dev/null"

max_open_files => 10000

#ignore_older => 0
}
}
filter {
json {
source => "url"
}
grok {
match => ["url", "^(?:https?://)?(?:www.)?(?[^/:]+)\S*"]
}

skip droped malicious to inser

#if "dropped_malicious" in [path] {

drop{}

}

change url type on the basis of directory

if "repute" in [path] and [t] == "b" {
mutate { replace => { "t" => "rb" } }
}

if "repute" in [path] and [t] == "d" {
mutate { replace => { "t" => "rd" } }
}

if "scan" in [path] and [t] == "b" {
mutate { replace => { "t" => "sb" } }
}

if "scan" in [path] and [t] == "d" {
mutate { replace => { "t" => "sd" } }
}

}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "timeseries"

document_id => "%{hostname}%{created_at}"

}

stdout { }
}

######################## API ####################################

PUT /_ilm/policy/timeseries_policy?pretty
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_docs": 1000
}
}
}
}
}
}

PUT /_index_template/timeseries_template?pretty
{
"index_patterns": ["timeseries-*"],
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"index.lifecycle.name": "timeseries_policy",
"index.lifecycle.rollover_alias": "timeseries"
}
}
}

PUT /timeseries-000001?pretty
{
"aliases": {
"timeseries": {
"is_write_index": true
}
}
}

GET /timeseries-*/_ilm/explain?format

POST /timeseries/_doc?pretty
{
"@timestamp": "1591890611",
"url": "http://2266608xyzshjd/register?key=1000350241",
"customer_name": "fareed",
"created_at": "2020-11-11 10:57:40.843094",
"hostname": "vm1-searcliteapi-v2",
"t": "m",
"s": "r",
"domain": "2266608xyzshjd"
}

############################# end #################################

Welcome to our community! :smiley:

Can you please reformat your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

Hi Warkolm,
thanks fro reply.

Now i am able to implement rollever api by setting the poll_interval": "1s" now it willl check againa and again after 1s and rollover index after reaching the limit max_doc condition.

PUT /_cluster/settings { "transient": { "indices.lifecycle.poll_interval": "1s" } }
but now i am facing issue that after creating multiple indices shortly i need to flush data then rollover work else it going to ingest data more then the defined max doc limit.
what is the automatic solution and after that i want to ingest data through log-stash. either it would be possible ther as well ?

Thanks

1 second is too small of a resolution for ILM to operate.

That might happen due tot he nature of distributed systems. But unless you have extremely small sizes does it matter?

for the testing purpose i have set this later i will set it to 1m when we move to production.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.