Facing challange during segregate the data from one index to another index

[Tek_Chand]

Hello Team,
We are using Akamai for CDN and WAF. We have configured datastream on akamai and getting those logs on our elasticsaerch. Please refer the below link for same:

Stream logs to Elasticsearch

As per doc we have created index on akamai dashboard and we have created index pattren for same. But Akamai doesn't support timestamp so its not creating new index for new date. So all the data is coming into single index and its creating issue when single indices size is growing.

We want to segregate the data on the basis of date and tries few things:

1. We have tried to configure the ILM:
   a). Created the ILM on the basis of age (1 Day)
   b). Created an index template and include that index in this template.
   c) Linked the template with ILM policy which we created.
   d) Created an alias for the index.
   e) Attach the ILM policy and alias with the index.

But its doing rollover operation and creating new index e.g akami-sf-00002 with size of 450byte but all the data exist in default index which we defined during akamai setup.

Rollover is creating new index but data can't came into this index because index name is static on akamai side.

2. Second option we have tried using reindex. Reindex copy the data into new index but its very slow process and we need to delete the data from source index once reindex is complete.

We want to acheive to create new index automatically on the basis of date or if any funtion is available in elasticsearch so we can segregate the data on the basis of date.

Can you please help me here?

Thank You

[leandrojmp (Leandro Pereira)]

What is the index that you set on the akamai side?

And what does your ILM and Templates looks like? Please share them using the preformatted text option, the </> button.

If you want to send the data directly to Elasticsearch you may have two approachs in this case, one is using Data Streams and following this documentation.

But I'm not sure this Akamai output would work with data streams.

To configure rollover using daily indices you would need to follow this part of the same documentation.

You basically need to create the ILM policy in the way you want to rollover after 1 day for example, then you need an index template where you need to have the index.lifecycle.name set to your ILM policy and index.lifecycle.rollover_alias set to an alias, something like akamai for example.

You will need to manually create the first indice, something like this:

PUT akamai-000001
{
  "aliases": {
    "akamai": {
      "is_write_index": true
    }
  }
}

Then on the Akamai side you will need to use this same alias as the index name.

[Tek_Chand]

@leandrojmp , Thank You for your quick response.

I was doing almost same steps except rollover_alias during index template creation.

PUT _index_template/timeseries_template
{
  "index_patterns": ["timeseries-*"],                 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "timeseries_policy",      
      "index.lifecycle.rollover_alias": "timeseries"    
    }
  }
}

Now i have made the changes and able to get response during lifecycle check progress.

GET akamai-sf-*/_ilm/explain

{
  "indices" : {
    "akamai-sf-01" : {
      "index" : "akamai-sf-01",
      "managed" : true,
      "policy" : "akamai-sf",
      "lifecycle_date_millis" : 1687236792712,
      "age" : "1.63h",
      "phase" : "hot",
      "phase_time_millis" : 1687236795050,
      "action" : "rollover",
      "action_time_millis" : 1687236796457,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1687236796457,
      "phase_execution" : {
        "policy" : "akamai-sf",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1687236168849
      }
    }
  }
}

Now when date will change rollover should work as per our expectation.
Will update you tomorrow morning.

Thank You