Using ILM with NetFlow module

[Ryne_Keel]

This is hard to explain, but index life cycle policies do not work with at least the logstash netflow module.

The module, or elasticsearch create a new index each day. They use the index template to do so. To get it to not error that:

illegal_argument_exception: index.lifecycle.rollover_alias [netflow-alias] does not point to index [netflow-2019.03.26-1]

You have to put an alias in the index template.

If you put the alias in the index template, then when it tries to rollover you get an error about duplicate alias in the template.

illegal_argument_exception: Rollover alias [netflow-alias] can point to multiple indices, found duplicated alias [[netflow-alias]] in index template [netflow]

If you go in at this point and remove the alias from the template (not the rollover alias, but the regular alias) and use index management to retry the lifecycle step, it will start creating rollover indexes, named as expected, however, nothing gets written to them and the system keeps writing into the index that was created by the module / index template at the beginning of the day.

Then, the next day when it creates the new index, you get the following error again because you removed the alias from the index template to even get to this point..

illegal_argument_exception: index.lifecycle.rollover_alias [netflow-alias] does not point to index [netflow-2019.03.26-1]

Problems persists in 6.7.0

[Ryne_Keel]

If anyone knows what I am doing wrong or if this is a bug, please help me understand how this is supposed to work.

[gbrown (Gordon Brown)]

I suspect this is an issue with the NetFlow module - in order for ILM to work correctly, Logstash will need to write to the netflow-alias alias, but from what you're describing it sounds like it's writing directly to concrete daily indices.

For a normal pipeline, I would direct you to the ILM section of the Elasticsearch output plugin configuration, but I don't see a way to change the index that the NetFlow module writes to. In the hopes of getting someone with more Logstash expertise looking at this, I'm going to move this topic to #logstash.

Regarding putting the alias in the template, we recommend that you don't do that - if data is correctly being written to the alias, rather than the underlying concrete index, what you should do is set up the index template, then create the first index with the alias by hand as described in the ILM Getting Started documentation. If the index has already been created, it is also possible to add the alias using the Index Aliases API.

[gbrown (Gordon Brown)]

I've moved this to #logstash from #elasticsearch and changed the title from "Elasticsearch: Index Rollover and Lifecycle Policy Issues" to "Using ILM with NetFlow module" as I believe that is more descriptive of the problem and more likely to get attention on the issue.

[jakelandis]

@Ryne_Keel thanks for raising this issue.

Logstash modules are not currently supported with ILM. I logged https://github.com/elastic/logstash/issues/10598 to track this issue.

[Ryne_Keel]

I will see if creating the index by hand makes a difference but need to look over the documentation and try it out before I will know if that helps. It would be awesome if it was an opt-in or opt-default as mentioned in your github issue, especially for netflow data. We aren't a huge ISP or anything, but have multiple transit providers and quite a few customers that use bandwidth, and we even generate 200+GB of netflow data per day.

Thanks guys!

[Ryne_Keel]

When I create the index manually / by hand it will write into the index I created, but when it rolls over I don't get anything to write into the rollover index and it keeps writing to the original one that I created.

What is odd is that I ran the following to manually make the index:

PUT netflow-2019.03.29-1
{
  "aliases": {
    "netflow-alias": {
      "is_write_index": true
    }
  }
}

Which creates netflow-2019.03.29-1 and shows

"lifecycle": {
        "name": "netflow_policy",
        "rollover_alias": "netflow-alias",
        "indexing_complete": "true"
      }

{
  "index.blocks.read_only_allow_delete": "false",
  "index.priority": "1",
  "index.query.default_field": [
    "*"
  ],
  "index.refresh_interval": "1s",
  "index.write.wait_for_active_shards": "1",
  "index.lifecycle.name": "netflow_policy",
  "index.lifecycle.rollover_alias": "netflow-alias",
  "index.lifecycle.indexing_complete": "true",
  "index.number_of_replicas": "1"
}

But the rolled over index ends up being named:

netflow-2019.03.29-000002