# Create a new indices for each random directory

**URL:** <https://discuss.elastic.co/t/create-a-new-indices-for-each-random-directory/357329>\
**Category:** Logstash\
**Tags:** reindex\
**Created:** [April 13, 2024, 5:05am UTC](https://discuss.elastic.co/t/create-a-new-indices-for-each-random-directory/357329 "2024-04-13T05:05:28Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sandeep\_Baljepally](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeep_baljepally/32/133479_2.png) [@Sandeep\_Baljepally](https://discuss.elastic.co/u/Sandeep_Baljepally)\
**Post date:** [April 13, 2024, 5:05am UTC](https://discuss.elastic.co/t/create-a-new-indices-for-each-random-directory/357329/1 "2024-04-13T05:05:28Z")

</div>

Hi Team, I have a requirement to create randomly generated directories from scripts to create separate indices i.e /mnt/data/logger/xyz/xyz-1.1.1.1 and /mnt/data/logger/xyz/xyz-1.2.1.1 here xyz-x.x.x.x these are generating programatically.

How can I create seperate indices for each random directory? Im using filebeat to collect the logs and logstash for filter and indexing.

exepectation: xyz-1.1.1.1-{date} and xyz-1.2.1.1-{date}

filebeat.yml:

```auto
- type: log
     enabled: true
     paths:
       - /mnt/data/logger/xyz/*/*.log
     fields:
       log_type: "xyz"

```

logstash:

````auto
  else if [fields][log_type] == "xyz" {
          mutate {
            add_field => { "index_name" => "xyz_logs" }
          }
        } ```
````

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 13, 2024, 5:37am UTC](https://discuss.elastic.co/t/create-a-new-indices-for-each-random-directory/357329/2 "2024-04-13T05:37:54Z")

</div>

> [@Sandeep\_Baljepally](#):
>
> How can I create seperate indices for each random directory?

That's simple enough. Parse [log][file][path] using grok to extract the directory name (e.g. see [this](https://discuss.elastic.co/t/is-it-possible-to-create-multiple-index-with-multiple-folder-name/321457/2) thread).

However, creating an index per day per directory can be a performance issue in elasticsearch because it can create a large number of small indexes/shards. (An index is stored in one or more shards.)

The [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html) recommends using multi-gigabyte shards.

---

<div class="post-metadata">

**Author:** ![Sandeep\_Baljepally](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeep_baljepally/32/133479_2.png) [@Sandeep\_Baljepally](https://discuss.elastic.co/u/Sandeep_Baljepally)\
**Post date:** [April 13, 2024, 6:09am UTC](https://discuss.elastic.co/t/create-a-new-indices-for-each-random-directory/357329/3 "2024-04-13T06:09:07Z")

</div>

could you please provide the snippet or update the above snippet for reference.  
filebeat:

```auto
- type: log
     enabled: true
     paths:
       - /mnt/data/logger/xyz/*/*.log
     fields:
       log_type: "xyz"

```

logstash

```auto
        else if [fields][log_type] == "xyz" {
          grok {
             match => { "[log][file][path]" => "%{GREEDYDATA:dir}/%{DATA:[@metadata][dest]}/%{GREEDYDATA}" }
           }
          mutate {
             add_field => { "index_name" => "xyz_logs-%{[@metadata][dest]}" }
           }
         }

```

Not working with above snippet

---

<div class="post-metadata">

**Author:** ![Sandeep\_Baljepally](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeep_baljepally/32/133479_2.png) [@Sandeep\_Baljepally](https://discuss.elastic.co/u/Sandeep_Baljepally)\
**Post date:** [April 13, 2024, 7:22am UTC](https://discuss.elastic.co/t/create-a-new-indices-for-each-random-directory/357329/4 "2024-04-13T07:22:45Z")

</div>

Thanks it worked.