# Indexing documents to elasticsearch monthly?
**URL:** https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793
**Category:** Logstash
**Created:** [August 24, 2016, 9:35am UTC](https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793 "2016-08-24T09:35:20Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@i5513](https://discuss.elastic.co/u/i5513)
#### Post date: [August 24, 2016, 9:35am UTC](https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793/1 "2016-08-24T09:35:20Z")
Hello,
We have a 3 node ELK cluster with 83000000 documents and 53 gb
Our cluster is populated by logstash who create a index by day
We have this scheme since 2014 january
Now elasticsearch is going slow. It have 484 index, 1 replica by document and 4832 shards
Using cluster.stats() from elasticsearch python took more than 10 seconds
kibana show time to time timeouts of 3 seconds and of 30 seconds
What do you recomend to fix these issues?
I'm thinking to switch from daily index to monthly index, changing it on logstash.
I imagine I should migrate all old index to this new scheme reindexing old indexes. Not sure
Thank you very much
PD: For reference I researched before writing this post on:
> [@Daily indices for a period of 1 month with wildcard search taking long time](https://discuss.elastic.co/t/daily-indices-for-a-period-of-1-month-with-wildcard-search-taking-long-time/56470):
>
> Hi Team, i am having 1TB of data over the span of 1 month where we create daily indices and currently we are creating around 20 daily indices with total data of 30-35 GB per day . Field-stats api is taking 50 secs to get the 1000 shards for the period of 1 month and our backend store is Network Storage . Can you please let us know if field stats will take 50 secs in this scenario from Kibana.
> [@HOW-TO create an index automatically per-month](https://discuss.elastic.co/t/how-to-create-an-index-automatically-per-month/46027):
>
> Good morning guys, Could you please possibly advise on how to achieve index management on a monthly basis ? For example, if the index alias is configured like { "actions": [ { "add": { "index": "{{ item }}", "index\_routing": "IXmac:cache", "search\_routing": "IXmac:cache", "alias": "TheApp" } }, { "add": { "index": "{{ item }}", "index\_…
> [@Create monthly indexes based on event attribute](https://discuss.elastic.co/t/create-monthly-indexes-based-on-event-attribute/29372/1):
>
> Hi All, can i perform JODA functions in the output section on other attributes then timestamp(%{+YYYY.MM})? i need to create indexes base on couple of attributes in my event and i can't use only the timestamp.
> [@30 billion unique documents (and counting)](https://discuss.elastic.co/t/30-billion-unique-documents-and-counting/23355/7):
>
> Running ES at scale is all about balance and sizing right. Like the 3 bears, not too big and not too small, just right. Big boxes will just be wasted and too small of boxes will have you hitting limits too soon. Given the way java works with heaps above 30GB-ish the best size for a node right now seems to be around 64GB RAM. More RAM will be utilized by the OS cache and definitely is a good thing but when you start to survey HW options and consider other factors like power, rack density an…
> [@Recommended way to reduce overload on ES](https://discuss.elastic.co/t/recommended-way-to-reduce-overload-on-es/17174/10):
>
> Good to know, thanks for the tip! Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 25 April 2014 10:09, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) wrote: Lucene is and has always been backwards-compatible. On the first merge of an index created with an earlier version it will get upgraded to the latest. And you can't always reindex (think huge installations), so I believe this is actually the intent of ES core t…
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 24, 2016, 11:00am UTC](https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793/2 "2016-08-24T11:00:15Z")
> Now elasticsearch is going slow. It have 484 index, 1 replica by document and 4832 shards
Woah. That's way too much. You must reduce the number of shards per index. Until your daily indexes reach a few tens of GB you shouldn't go beyond one shard per index.
---
### Author:  [@i5513](https://discuss.elastic.co/u/i5513)
#### Post date: [August 25, 2016, 6:47am UTC](https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793/3 "2016-08-25T06:47:43Z")
Hello @magnusbaeck,
I have resumed my indices they store logs from logstash.
Currently I recollect in the first index less than 1 gb by month, so I guess I can reshard from daily to monthly such index. And I suppose I can change from daily to monthly the creation of indices.
At the second index, we are currently storing only 1 month of logs (73000000 docs / 41 GB), I'm going to change to 1 shard on such index
My plan of such operation is:
1. Make snapshot of indices
2. Create one index with one shard by each month where we recollected data
3. Fill such index with daily index data, with elasticdump utility
4. Remove old daily indices
Do you think it is a good plan ? Would you have a better alternative? Would you recomend to skip monthly created indices and go with daily one ?
Thank you very much
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 25, 2016, 7:20am UTC](https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793/4 "2016-08-25T07:20:15Z")
That looks like a reasonable plan. I generally prefer daily indexes because a) correcting mapping mistakes is much faster and b) you can clean up older indexes with a higher resolution.
---
### Author:  [@i5513](https://discuss.elastic.co/u/i5513)
#### Post date: [August 26, 2016, 12:35pm UTC](https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793/5 "2016-08-26T12:35:25Z")
Thank you @magnusbaeck,
I scripted it, to httpd indice I prefer to wait, because I changed the config to generate new indices with only one shard (+ replica)
Here is the script (in spanish):
```
ano_actual=$(date +%Y)
mes_actual=$(date +%m)
if [-z "$1"]
then
echo "Uso: $0 [indice]"
echo "Ejemplos:
$0 logstash
else
indice="$1"
fi
curl -o /tmp/indices.txt -s -x '' 'http://localhost:9200/_cat/indices/'"$indice"'*'
if [-n "$2"]
then
meses="$2"
else
meses=$(grep $indice- /tmp/indices.txt |
sed -n 's,.*-\(....\)\.\(..\)\....*,\1.\2,p' |
grep -v "$ano_actual\.$mes_actual" | sort -u)
fi
for m in $meses
do
dias=$(grep -F $indice-$m. /tmp/indices.txt | awk '{print $3}')
if [-n "$dias"]
then
echo "Migrando $m ..."
curl -X PUT -x '' http://localhost:9200/$indice-$m &> /var/log/elasticsearch/reshard/creacion-$m.log
fi
for d in $dias
do
echo "Pasando $indice-$d ..."
elasticdump --input http://localhost:9200/$d --output http://localhost:9200/$indice-$m --limit 10000 &> /var/log/elasticsearch/reshard/reunificacion-$d.log
if [$? = 0]
then
curl -X DELETE -x '' http://localhost:9200/$d &> /var/log/elasticsearch/reshard/eliminacion-$d.log
fi
done
done
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:41am UTC](https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793/6 "2017-07-06T04:41:27Z")