Indices management from multiple logstash instances

[manfred]

Hi,
I'm testing ELK currently. I have the following architecture:

Logstash has three instances running in three different data centers

• logx01
• logx02
• logx03

Elasticsearch cluster with 4 nodes:

• es01 / data node, hot
• es02 / data node, hot
• es03 / data node, hot
• es04 / data node, warm

Configuration applied:

PUT _ilm/policy/policy_of_truth   
 {
   "policy": {                       
   "phases": {
   "hot": {                      
   "actions": {
   "rollover": {             
     "max_size": "7GB",
     "max_age": "10m"
    }
   }
  },
  "delete": {
    "min_age": "20m",           
    "actions": {
      "delete": {}              
      }
    }
  }
}

}

PUT _template/logx_template
{
  "index_patterns": ["logx*"],                 
  "settings": {
  "number_of_shards": 10,
  "number_of_replicas": 1,
  "index.lifecycle.name": "policy_of_truth",      
  "index.lifecycle.rollover_alias": "logx"    
  }
}

PUT logx-000001
{
   "aliases": {
   "logx": {
    "is_write_index": true
   }
 }
}

My main concern now is to make a rollover policy to work, i.e. from hot to delete phase. I tested it with fixed index name in logstash like "logx01-000001" for example. It works ok from one logstash instance, but my main problem starts when I add other two logstash instances.

I want to have only one index from three logstashes. That assumes that the index must have the same name on three instances, like "logx-000001". This setup doesn't work for me. Transition from hot to delete phase doesn't happen, and all documents are stored only in one index (logx-000001), all others have 0 docs. I'm highly sure that is because I wrongly understand how rollover alias work. Besides get the following error in the Index Management: illegal_argument_exception: index.lifecycle.rollover_alias [logx] does not point to index [logx-000001]. Thanks for any advice.