How to write to currently managed index from logstash (via Elasticsearch output)

Elastic Stack Logstash
1

My current deployment setup is as below.

I want to write to filebeat indices with hostnames while ILM is in place to manage the indices

EG:
filebeat-7.2.0-host-name-1-000001

In order to achieve this,
I'm trying to manage ilm on my own,
Reason for chosing this path is logstash does not support dynamic index names when we ask it to do the ILM for us

  1. I have an ILM policy created as below
{
  "filebeat-7.2.0" : {
    "version" : 11,
    "modified_date" : "2019-07-30T17:19:35.059Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "100mb",
              "max_age" : "1h"
            }
          }
        },
        "delete" : {
          "min_age" : "365d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "7d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "warm"
              }
            }
          }
        }
      }
    }
  }
}
  1. then I create a template per host as below manually via api

{
  "filebeat-7.2.0-host-name-1" : {
    "order" : 1,
    "index_patterns" : [
      "filebeat-7.2.0-host-name-1*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "filebeat-7.2.0",
          "rollover_alias" : "filebeat-host-name-1"
        },
.........................
..................................................
    "aliases" : { }
  }
  1. Then in-order to initiate the index i create the first index with a json similar to this
> curl -XPUT -H 'Content-Type: application/json' https://XXXXXXXXXXXXXXXXXXXXXXXXX.aws.found.io:9243/filebeat-7.2.0-host-name-1 -d@index.json -u XXXXXXX:XXXXXXXXXXXXXXXXXXXXXXX
 
### index.json
{
  "aliases": {
    "filebeat-{{ ansible_hostname | lower }}":{
    # EG:  "filebeat-host-name-1":{
      "is_write_index": true
    }
  }
}
  1. Then my logstash pipeline looks like below
output {
  if [@metadata][pipeline] {
    elasticsearch {
      ssl => true
      hosts => ["XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.aws.found.io:9243"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[host][hostname]}-000001"
      pipeline => "%{[@metadata][pipeline]}"
      user => XXXXXXXXXXX
      password => "XXXXXXXXXXX"
      ilm_enabled => false
    }
  }
  else {
    elasticsearch {
      ssl => true
      hosts => ["XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.aws.found.io:9243"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[host][hostname]}-000001"
      user => XXXXXXXXXXXXXXX
      password => "XXXXXXXXXXX"
      ilm_enabled => false
    }
  }
}
  1. After all this, when I start pushing data to ES via logstash, it initially works fine.....
    ILM rotation happens without issues........

BUT............

  1. After ILM rotates the index,
filebeat-7.2.0-host-name-1-000001 <-- this becomes not writable
filebeat-7.2.0-host-name-1-000002 <-- this becomes the currently managed index, and this is the only writable index for the alias
  1. But since Logstash is always writing to an index like this "filebeat-7.2.0-host-name-1-000001"

If there is anyway that i can write to currently managed index from logstash output,
The last piece in this puzzle will be fixed...........

2

This is how my alias looks like after rotation,

GET /_alias/filebeat-host-name-1
{
  "filebeat-7.2.0-host-name-1-000001" : {
    "aliases" : {
      "filebeat-host-name-1" : {
        "is_write_index" : false
      }
    }
  },
  "filebeat-7.2.0-host-name-1-000002" : {
    "aliases" : {
      "filebeat-host-name-1" : {
        "is_write_index" : true
      }
    }
  }
}
3

In the elasticsearch output you should set the index to the alias

index => "filebeat-host-name-1"
4

Thanks @Badger looks like this did the trick,
i'll let it rotate my indices and mark your reply as the solution

5

Thanks again, this did the trick

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.