# Default Index Pattern (logstash-yyyy.MM.dd) Failing

**URL:** <https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195>\
**Category:** Logstash\
**Created:** [September 26, 2022, 5:46pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195 "2022-09-26T17:46:01Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![scriner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scriner/32/59203_2.png) [@scriner](https://discuss.elastic.co/u/scriner)\
**Post date:** [September 26, 2022, 5:46pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195/1 "2022-09-26T17:46:01Z")

</div>

Hi all. We are using the following pipeline config file to ship syslogs to Logstash:

```auto
input {
  tcp {
    port => 1514
    codec => "json"
    type => "syslog"
  }
  udp {
    port => 1514
    codec => "json"
    type => "syslog"
  }
}

filter {
  # This replaces the host field (UDP source) with the host that generated the message (sysloghost)
  if [sysloghost] {
    mutate {
      replace => ["host", "%{sysloghost}"]
      remove_field => "sysloghost" # prune the field after successfully replacing "host"
    }
  }
}

output {
  elasticsearch {
    hosts => ["{IP}:9200","{IP}:9200","{IP}:9200","{IP}:9200"]
    index => "logstash-%{+yyyy.MM.dd}"
  }
}

```

This works, but the index is not getting tagged correctly when we look at the documents in Kibana. If I change the index line in the output block to something else - `syslog-yyyy.MM.dd`, `logstash-yyyy.MM.dda`, etc. - then everything is fine and the indices are created with the current date as you'd expect. If I leave them as shown with the standard `logstash-yyyy.MM.dd` pattern, it is not respected. Right now it's just using an old date with `-00001` appended to the end, which I'm guessing is a rollover index or something.

I guess we can just use a different index pattern like syslog like I mentioned above, but I really would prefer to know how to fix this.

Thanks!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 26, 2022, 5:49pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195/2 "2022-09-26T17:49:21Z")

</div>

> [@scriner](#):
>
> `index => "logstash-%{+yyyy.MM.dd}"`

If elasticsearch supports it (i.e. a recent version) then logstash will be using [ILM](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm) and the index option is ignored.

---

<div class="post-metadata">

**Author:** ![scriner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scriner/32/59203_2.png) [@scriner](https://discuss.elastic.co/u/scriner)\
**Post date:** [September 26, 2022, 7:41pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195/3 "2022-09-26T19:41:08Z")

</div>

I appreciate the reply - I think I've wandered down this path in the past but I must not understand completely.

When changing the Index Lifecycle Policy for the `logs` Managed Index Template (which is the closest thing I can find that might apply to my current predicament) from 30 days or 50GB to 1 day, the index remains `logstash-2022.03.29-000001`. And, if I understand right, this policy would have already rolled this index over a long time ago based on the 30 day policy. I don't understand how we've been using the same index for 6 months.

Ideally, today for example, each document sent would get tagged with index `logstash-2022.09.26`, and increment with the days. Is this not possible using the default `logstash-yyyy.MM.dd` format?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 26, 2022, 11:50pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195/4 "2022-09-26T23:50:43Z")

</div>

What version of logstash and Elasticsearch are you using?

And if I read it correctly you want daily indices? Is that correct?

And are you planning to delete them after a certain length of time?

---

<div class="post-metadata">

**Author:** ![scriner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scriner/32/59203_2.png) [@scriner](https://discuss.elastic.co/u/scriner)\
**Post date:** [September 27, 2022, 12:49pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195/5 "2022-09-27T12:49:06Z")

</div>

We're currently at version 7.17.6.

And yes, that was what I wanted originally, but I think I've decided to just let ILM manage all of these for me and use the rollover pattern naming. I think the only reason we wanted daily indices is to be able to reliably expire them after they're a year old, but my understanding is that ILM can handle that using document data so I think doing it this way is fine.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 27, 2022, 1:06pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195/6 "2022-09-27T13:06:01Z")

</div>

> [@scriner](#):
>
> but my understanding is that ILM can handle that using document data so I think doing it this way is fine.

The ILM will use the date in the index name as the creation date, for example for the index `logstash-2022.03.29-000001` it will use `2022.03.29` as the date to calculate when it will move between the phases in the ILM policy.

---

<div class="post-metadata">

**Author:** ![scriner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scriner/32/59203_2.png) [@scriner](https://discuss.elastic.co/u/scriner)\
**Post date:** [September 27, 2022, 1:26pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195/7 "2022-09-27T13:26:24Z")

</div>

I'm seeing our index come in as this: `logstash-ilm-000001`

Using this output block:

```auto
output {
  elasticsearch {
    hosts => ["{CLUSTER}"]
    ilm_enabled => true
    ilm_rollover_alias => "logstash-ilm"
    ilm_policy => "AnnualRotateRolling"
  }
}

```

Trying to set `ilm_pattern` as `{now/d}-000001` as shown in the docs stops our logs from coming in.

---

<div class="post-metadata">

**Author:** ![scriner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scriner/32/59203_2.png) [@scriner](https://discuss.elastic.co/u/scriner)\
**Post date:** [September 27, 2022, 1:33pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195/8 "2022-09-27T13:33:43Z")

</div>

This is now working as expected - removing the `ilm_rollover_alias` line allowed the index to include the date as expected. Thanks all!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 25, 2022, 1:33pm UTC](https://discuss.elastic.co/t/default-index-pattern-logstash-yyyy-mm-dd-failing/315195/9 "2022-10-25T13:33:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.