Logstash and ILM rollover

[nikita2206 (Nikita Nefedov)]

Every tutorial or manual out there talks about integrating Filebeat with ILM rollover. We're not using Filebeat (it doesn't really make sense to send logs through actual files when we could do that without involving filesystem by going this route: docker container -> logspout -> logstash -> elastic).

In Logstash, we're determining what kind of index the document/log event belongs to (there are app logs, access-logs and unsorted logs; the idea being that we want to retain app logs for the longest, access-logs for as much as can be fit in 20gb and unsorted can only enjoy 4gb of space). I suppose in Logstash I can use the index alias directly, so I've defined index alias for access logs with the name access-logs in the Index Template by:

• Creating a lifecycle that manages 20gb rollover, called access-logs
• Adding "index": { "lifecycle": { "name": "access-logs", "rollover_alias": "access-logs" } in the settings of Index Template

This didn't seem to work. When I ran GET /_alias to get all defined aliases, although I could see access-logs, it differed from other aliases like apm-7.7.0-span one for example, the latter one looked like this:

  "apm-7.7.0-span-000001" : {
    "aliases" : {
      "apm-7.7.0-span" : {
        "is_write_index" : true
      }
    }
  }

Mine looked like this:

  "app-000001" : {
    "aliases" : {
      "app" : { }
    }
  }

I then tried adding

{
  "access-logs": { "is_write_index": true }
}

to the Index Template in Aliases section. This seemed to do the job but on the next day I saw an error saying that the alias points to two indexes at the same time (both indexes are actually the same index as well).

illegal_argument_exception: Rollover alias [access-logs] can point to multiple indices, found duplicated alias [[access-logs]] in index template [access-logs]

I suppose what I did basically configured the alias twice.
So what's the correct way to configure ILM in Kibana for indexes without any involvement of Filebeats? While letting Logstash use a single index name (an alias) to send documents to.

Update: I've removed the {"access-logs": {"is_write_index": true } } from Aliases section and ILM policy from Settings in the Index Template, then in the ILM I assigned the lifecycle to that index template (ie using GUI instead of writing JSON settings manually), then deleted the index itself so that it's recreated with new configuration. In order to test that it will actually do a rollover I've set ILM hot phase threshold to 100mb, after doing that I've started getting this error message:

illegal_argument_exception: index.lifecycle.rollover_alias [access-logs] does not point to index [access-logs-000001]

I tried to kill that index again so that it's recreated, the error still remains. GET /_alias says:

  "access-logs-000001" : {
    "aliases" : { }
  },

I don't understand what's happening.

[nikita2206 (Nikita Nefedov)]

Documenting my findings for others who might have the same issue:

I've finally found relevant documentation here https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_writing_to_different_indices_best_practices
I thought that aliases are fully handled on the Kibana and Elastic side, and clients only need to use alias in place of the index name and Elasticsearch will automatically replace the alias with the correct index name. Turns out that when the index doesn't exist yet, it's Logstash that is supposed to create it, and for it to do that correctly it's not enough to just point its index to the alias name, you have to also configure ILM support in Logstash. And Logstash doesn't support multiple indexes with ILM, because to make multiple indexes work in the same Logstash pipeline you have to use dynamic interpolation of the index name parameter: index => "%{[@metadata][index]}" (where you populate [@metadata][index] somewhere above in the pipeline). And Logstash + ILM does not support interpolation for index names...

They suggest defining multiple elasticsearch outputs, one per alias. But that comes with its problems and frankly I have no idea how to do that and there doesn't seem to documentation for that or it's too hard to find. My next step will be using Fluentd as a replacement to Logstash which should support multiple Elastic targets better than Logstash does, and I also wasn't happy with Logstash eating 500mb of memory just to ship logs.