# How to write to currently managed index from logstash (via Elasticsearch output)

**URL:** <https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183>\
**Category:** Logstash\
**Created:** [July 31, 2019, 4:40pm UTC](https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183 "2019-07-31T16:40:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Thanura\_Kannangara](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\
**Post date:** [July 31, 2019, 4:40pm UTC](https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183/1 "2019-07-31T16:40:51Z")

</div>

My current deployment setup is as below.

I want to write to filebeat indices with hostnames while ILM is in place to manage the indices

EG:  
filebeat-7.2.0-host-name-1-000001

In order to achieve this,  
I'm trying to manage ilm on my own,  
Reason for chosing this path is logstash does not support dynamic index names when we ask it to do the ILM for us

1. I have an ILM policy created as below

```auto
{
  "filebeat-7.2.0" : {
    "version" : 11,
    "modified_date" : "2019-07-30T17:19:35.059Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "100mb",
              "max_age" : "1h"
            }
          }
        },
        "delete" : {
          "min_age" : "365d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "7d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "warm"
              }
            }
          }
        }
      }
    }
  }
}

```

1. then I create a template per host as below manually via api

```auto

{
  "filebeat-7.2.0-host-name-1" : {
    "order" : 1,
    "index_patterns" : [
      "filebeat-7.2.0-host-name-1*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "filebeat-7.2.0",
          "rollover_alias" : "filebeat-host-name-1"
        },
.........................
..................................................
    "aliases" : { }
  }

```

1. Then in-order to initiate the index i create the first index with a json similar to this

```auto
> curl -XPUT -H 'Content-Type: application/json' https://XXXXXXXXXXXXXXXXXXXXXXXXX.aws.found.io:9243/filebeat-7.2.0-host-name-1 -d@index.json -u XXXXXXX:XXXXXXXXXXXXXXXXXXXXXXX
 
### index.json
{
  "aliases": {
    "filebeat-{{ ansible_hostname | lower }}":{
    # EG: "filebeat-host-name-1":{
      "is_write_index": true
    }
  }
}

```

1. Then my logstash pipeline looks like below

```auto
output {
  if [@metadata][pipeline] {
    elasticsearch {
      ssl => true
      hosts => ["XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.aws.found.io:9243"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[host][hostname]}-000001"
      pipeline => "%{[@metadata][pipeline]}"
      user => XXXXXXXXXXX
      password => "XXXXXXXXXXX"
      ilm_enabled => false
    }
  }
  else {
    elasticsearch {
      ssl => true
      hosts => ["XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.aws.found.io:9243"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[host][hostname]}-000001"
      user => XXXXXXXXXXXXXXX
      password => "XXXXXXXXXXX"
      ilm_enabled => false
    }
  }
}

```

1. After all this, when I start pushing data to ES via logstash, it initially works fine.....  
ILM rotation happens without issues........

BUT............

1. After ILM rotates the index,

```auto
filebeat-7.2.0-host-name-1-000001 <-- this becomes not writable
filebeat-7.2.0-host-name-1-000002 <-- this becomes the currently managed index, and this is the only writable index for the alias

```

1. But since Logstash is always writing to an index like this "filebeat-7.2.0-host-name-1-000001"

**If there is anyway that i can write to currently managed index from logstash output,  
The last piece in this puzzle will be fixed...........**

---

<div class="post-metadata">

**Author:** ![Thanura\_Kannangara](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\
**Post date:** [July 31, 2019, 5:16pm UTC](https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183/2 "2019-07-31T17:16:15Z")

</div>

This is how my alias looks like after rotation,

```auto
GET /_alias/filebeat-host-name-1
{
  "filebeat-7.2.0-host-name-1-000001" : {
    "aliases" : {
      "filebeat-host-name-1" : {
        "is_write_index" : false
      }
    }
  },
  "filebeat-7.2.0-host-name-1-000002" : {
    "aliases" : {
      "filebeat-host-name-1" : {
        "is_write_index" : true
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 31, 2019, 5:28pm UTC](https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183/3 "2019-07-31T17:28:56Z")

</div>

In the elasticsearch output you should set the index to the alias

```
index => "filebeat-host-name-1"
```

---

<div class="post-metadata">

**Author:** ![Thanura\_Kannangara](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\
**Post date:** [July 31, 2019, 5:36pm UTC](https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183/4 "2019-07-31T17:36:11Z")

</div>

Thanks @Badger looks like this did the trick,  
i'll let it rotate my indices and mark your reply as the solution

---

<div class="post-metadata">

**Author:** ![Thanura\_Kannangara](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\
**Post date:** [July 31, 2019, 6:16pm UTC](https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183/5 "2019-07-31T18:16:12Z")

</div>

Thanks again, this did the trick

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 28, 2019, 6:16pm UTC](https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183/6 "2019-08-28T18:16:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.