# Split data into multiple indices automatically on insert

**URL:** <https://discuss.elastic.co/t/split-data-into-multiple-indices-automatically-on-insert/39650>\
**Category:** Elasticsearch\
**Created:** [January 20, 2016, 12:35pm UTC](https://discuss.elastic.co/t/split-data-into-multiple-indices-automatically-on-insert/39650 "2016-01-20T12:35:41Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![edeak](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@edeak](https://discuss.elastic.co/u/edeak)\
**Post date:** [January 20, 2016, 12:35pm UTC](https://discuss.elastic.co/t/split-data-into-multiple-indices-automatically-on-insert/39650/1 "2016-01-20T12:35:41Z")

</div>

Hi guys,

what I like to do is that when I insert my data which has a timestamp field, my data is automatically should be splitted across weekly indices and if it's necessary, a new index with pre-defined mapping should be created.

Is there a way to do that whith ES config or I have to do it in business logic outside elasticsearch?

---

<div class="post-metadata">

**Author:** ![tommoholmes10](https://avatars.discourse-cdn.com/v4/letter/t/ed8c4c/32.png) [@tommoholmes10](https://discuss.elastic.co/u/tommoholmes10)\
**Post date:** [January 20, 2016, 2:12pm UTC](https://discuss.elastic.co/t/split-data-into-multiple-indices-automatically-on-insert/39650/2 "2016-01-20T14:12:43Z")

</div>

If you use multiple node in server, elasticsearch automatically split, you only configure same cluster\_name

---

<div class="post-metadata">

**Author:** ![edeak](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@edeak](https://discuss.elastic.co/u/edeak)\
**Post date:** [January 20, 2016, 3:03pm UTC](https://discuss.elastic.co/t/split-data-into-multiple-indices-automatically-on-insert/39650/3 "2016-01-20T15:03:33Z")

</div>

Thanks for your reply. Maybe you misunderstood me.

Lets say I have data from 2016.01.01 to 2016.01.20.  
When I start to insert my data I want that ES creates an index named mytype-20160101 and it stores all the data where the timestamp field is less than 2016.01.08. 00:00:00.  
Next, it should create a new index named mytype-20160108 and it stores all the data where the timestamp field is between 2016.01.08 00:00:00 and 2016.01.15 00:00:00  
... and so on.

If I use logstash for inserting data, can I achieve this?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 20, 2016, 7:42pm UTC](https://discuss.elastic.co/t/split-data-into-multiple-indices-automatically-on-insert/39650/4 "2016-01-20T19:42:21Z")

</div>

> When I start to insert my data I want that ES creates an index named mytype-20160101 and it stores all the data where the timestamp field is less than 2016.01.08. 00:00:00.

ES won't do this automatically. You have to tell it which index to store a new document.

> If I use logstash for inserting data, can I achieve this?

Yes.

---

<div class="post-metadata">

**Author:** ![edeak](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@edeak](https://discuss.elastic.co/u/edeak)\
**Post date:** [January 21, 2016, 9:00am UTC](https://discuss.elastic.co/t/split-data-into-multiple-indices-automatically-on-insert/39650/5 "2016-01-21T09:00:37Z")

</div>

Thanks for your reply Magnus. I'm trying to import my data with logstash, and I'm trying to import my data into monthly created indexes. I have a date field called createDate and thats what I want to use.

Here's my config:

```
input {
    jdbc {        
        jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"        
        jdbc_user => "user"
		jdbc_password => "pass"        
        jdbc_driver_library => "mysql-connector-java-5.1.38-bin.jar"        
        jdbc_driver_class => "com.mysql.jdbc.Driver"
		jdbc_paging_enabled => "true"
		jdbc_page_size => "2"
        # just for testing
        statement => "SELECT * FROM mytable skip 0 limit 10"
    }
}

output {
    elasticsearch {
		hosts => ["localhost:9200"]		
		# here's the trick...
		index => "my-index-{myformatteddate}"
		document_type => "mydoc"
	}
}

```

What filter should I apply and how, to create a new field myformatteddate which contains the year and month of the createDate?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 21, 2016, 6:19pm UTC](https://discuss.elastic.co/t/split-data-into-multiple-indices-automatically-on-insert/39650/6 "2016-01-21T18:19:56Z")

</div>

One typically uses the date filter to parse date strings and convert them to ISO8601 format in UTC. Then you can use the `%{+YYYY.MM.dd}` notation in the index name to have Logstash insert the date from the `@timestamp` field into the index name. Exactly what the date filter should look like depends on what the `createDate` field looks like. I haven't used the jdbc input so I'm not sure what happens with date fields. Use a `stdout { codec => rubydebug }` until you get what you want. Don't jump into Elasticsearch prematurely.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:22pm UTC](https://discuss.elastic.co/t/split-data-into-multiple-indices-automatically-on-insert/39650/7 "2017-07-05T23:22:25Z")

</div>