ES + filebeat index lifecycle management issue

[bhkybbzl (Jeremy Brown)]

Hi,

I've got a confusing issue when using index lifecycle management feature in ES and filebeat.

I'm collecting logs from various pods in K8S using filebeat, and sending them to Elasticsearch, I have a few requirements listed below:

1. index pattern should be log-{k8s.namespace}-{time}, where k8s.namespace is the kubernetes namespace of the pod the log comes from. Therefore different log might have different index in ES
2. All the fields in logs should be treated as keyword in ES so that they could be searched as a whole, so I have a template.json used for index template listed below
3. the index should be automatically deleted after 7 days from creation. (where ILM is used for)

so I have a filebeat.yml configuration like this (irrelevant fields are omitted):

setup.template.overwrite: true
setup.template.json.enabled: true
setup.template.json.path: "template.json"
setup.template.json.name: "log"

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "log-rollover"
setup.ilm.pattern: "-000001"

output.elasticsearch:
  indices:
    - index: "%{[kubernetes.namespace]}-%{+yyyy.MM.dd}"
      when.has_fields:
        - "kubernetes.namespace"
    - index: "default-%{+yyyy.MM.dd}"

and the template.json file: (irrelevant fields omitted)

{
"order": 0,
"index_patterns": [
	"log-*"
],
"settings": {
	"index": {
		"number_of_shards": "7",
		"number_of_replicas": "1",
		"refresh_interval": "60s",
        "lifecycle.name": "log-policy",
        "lifecycle.rollover_alias": "log-rollover"
	}
}

I've also got an ILM policy already set in ES, which has the name log-policy corresponding to the name above.

After I set all these, I got the error:

illegal_argument_exception: index.lifecycle.rollover_alias [log-rollover] does not point to index

I know this is because the alias should be created before the first index.

So my question is:

1. in my filebeat.yml, there is setup.ilm.rollover_alias: "log-rollover", why doesn't it create the alias for me? If it doesn't what else config can ?
2. what is setup.ilm.pattern used for? It seems this config doesn't control the index pattern sent to ES cuz I didn't see this pattern in ES.
3. I'm aware that if setup.ilm.enabled: auto is set, the indices might be ignored, so in this situation (ILM enabled), what config should I use to set flexible index pattern like indices? I need those two indices in case some logs might not have a kubernetes.namespace field.
4. in both template.json and filebeat.yml there can be setup.ilm.rollover_alias and setup.ilm.name configs, if both set, which one would be used ?

can anyone help me...please.. I've been struggling for a few days already...