# Getting Data To Persist

**URL:** <https://discuss.elastic.co/t/getting-data-to-persist/3078>\
**Category:** Elasticsearch\
**Created:** [July 7, 2010, 11:16pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078 "2010-07-07T23:16:30Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [July 7, 2010, 11:16pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/1 "2010-07-07T23:16:30Z")

</div>

Hi,

I installed elasticsearch-0.8.0 on 64-bit Ubuntu 9.10 system. It comes  
up, runs fine, creates the indices I want, etc., but has a very serious  
problem I can't trace down. If I do a 'bin/service/elasticsearch  
restart' command (change config/restart), the entire database  
disappears. I'm fairly certain this is not the desired action. Here's  
a console log showing what happens:

ken@stealth:/opt/elasticsearch-0.8.0$ ll work/elasticsearch/indices/  
total 4  
drwxr-xr-x 22 ken ken 4096 2010-07-07 17:48  
74f73a5f-50a0-4ad4-a574-6466300b9399

\<go off and run ES for a while, then restart\>

ken@stealth:/opt/elasticsearch-0.8.0$ bin/service/elasticsearch restart  
Stopping ElasticSearch...  
Stopped ElasticSearch.  
Starting ElasticSearch...Waiting for ElasticSearch......  
ElasticSearch started.

ken@stealth:/opt/elasticsearch-0.8.0$ ll  
work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/  
ls: cannot access  
work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/: No  
such file or directory  
ken@stealth:/opt/elasticsearch-0.8.0$ ll work/elasticsearch/indices/  
total 0

I've read through the docs many times (very terse), changed config,  
blown it away and unzipped again, nothing helps. Perhaps you can tell  
me what I can do to correct the problem. I'm pretty sure it should  
persist the data.

...Thanks,  
...Ken

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 8, 2010, 2:09pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/2 "2010-07-08T14:09:47Z")

</div>

Have you configured a gateway for elasticsearch? The configuration is  
simple:

gateway:  
type: fs  
fs:  
location: /path/to/shared/location

-shay.banon

On Thu, Jul 8, 2010 at 2:16 AM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> Hi,
> 
> I installed elasticsearch-0.8.0 on 64-bit Ubuntu 9.10 system. It comes  
> up, runs fine, creates the indices I want, etc., but has a very serious  
> problem I can't trace down. If I do a 'bin/service/elasticsearch  
> restart' command (change config/restart), the entire database  
> disappears. I'm fairly certain this is not the desired action. Here's  
> a console log showing what happens:
> 
> ken@stealth:/opt/elasticsearch-0.8.0$ ll work/elasticsearch/indices/  
> total 4  
> drwxr-xr-x 22 ken ken 4096 2010-07-07 17:48  
> 74f73a5f-50a0-4ad4-a574-6466300b9399
> 
> \<go off and run ES for a while, then restart\>
> 
> ken@stealth:/opt/elasticsearch-0.8.0$ bin/service/elasticsearch restart  
> Stopping Elasticsearch...  
> Stopped Elasticsearch.  
> Starting Elasticsearch...Waiting for Elasticsearch......  
> Elasticsearch started.
> 
> ken@stealth:/opt/elasticsearch-0.8.0$ ll  
> work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/  
> ls: cannot access  
> work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/: No  
> such file or directory  
> ken@stealth:/opt/elasticsearch-0.8.0$ ll work/elasticsearch/indices/  
> total 0
> 
> I've read through the docs many times (very terse), changed config,  
> blown it away and unzipped again, nothing helps. Perhaps you can tell  
> me what I can do to correct the problem. I'm pretty sure it should  
> persist the data.
> 
> ...Thanks,  
> ...Ken

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [July 12, 2010, 5:40pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/3 "2010-07-12T17:40:40Z")

</div>

Thanks! One more learning hurdle down.

Is there a list somewhere of all the configuration items that I could  
adjust? I need to increase the number of indices that I can build on a  
node. I'm getting 500 errors every time I add a new index or mapping.

As I understand it, given the limit of 100 shards per node, and the  
default of 5 shards per index, each node can support only 20 indices.  
Am I missing something?

Or, is it possible to run more than one node per machine for testing?

I need to be able to run 100+ indices with 10+ mappings per index, all  
fairly low use. One machine should be able to handle the load for  
testing.

...Ken

Shay Banon wrote:

> Have you configured a gateway for elasticsearch? The configuration is  
> simple:
> 
> gateway:  
> type: fs  
> fs:  
> location: /path/to/shared/location
> 
> -shay.banon
> 
> On Thu, Jul 8, 2010 at 2:16 AM, Kenneth Loafman  
> \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> ```
> Hi,
> 
> I installed elasticsearch-0.8.0 on 64-bit Ubuntu 9.10 system. It comes
> up, runs fine, creates the indices I want, etc., but has a very serious
> problem I can't trace down. If I do a 'bin/service/elasticsearch
> restart' command (change config/restart), the entire database
> disappears. I'm fairly certain this is not the desired action. Here's
> a console log showing what happens:
> 
> ken@stealth:/opt/elasticsearch-0.8.0$ ll work/elasticsearch/indices/
> total 4
> drwxr-xr-x 22 ken ken 4096 2010-07-07 17:48
> 74f73a5f-50a0-4ad4-a574-6466300b9399
> 
> <go off and run ES for a while, then restart>
> 
> ken@stealth:/opt/elasticsearch-0.8.0$ bin/service/elasticsearch restart
> Stopping ElasticSearch...
> Stopped ElasticSearch.
> Starting ElasticSearch...Waiting for ElasticSearch......
> ElasticSearch started.
> 
> ken@stealth:/opt/elasticsearch-0.8.0$ ll
> work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/
> ls: cannot access
> work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/: No
> such file or directory
> ken@stealth:/opt/elasticsearch-0.8.0$ ll work/elasticsearch/indices/
> total 0
> 
> I've read through the docs many times (very terse), changed config,
> blown it away and unzipped again, nothing helps. Perhaps you can tell
> me what I can do to correct the problem. I'm pretty sure it should
> persist the data.
> 
> ...Thanks,
> ...Ken
> 
> ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 12, 2010, 6:08pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/4 "2010-07-12T18:08:58Z")

</div>

Actually, the 100 shards per node is just arbitrary limit, and to be honest,  
not properly designed in terms of configuration (let me see what I can do  
for 0.9 regarding that). Second, creating so many indices / shards is going  
to be pretty heavy on a single node. In 0.9, that overhead is going to be  
lower, but still... . And yes, you can certainly start more than one node on  
a single machine.

-shay.banon

On Mon, Jul 12, 2010 at 8:40 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> Thanks! One more learning hurdle down.
> 
> Is there a list somewhere of all the configuration items that I could  
> adjust? I need to increase the number of indices that I can build on a  
> node. I'm getting 500 errors every time I add a new index or mapping.
> 
> As I understand it, given the limit of 100 shards per node, and the  
> default of 5 shards per index, each node can support only 20 indices.  
> Am I missing something?
> 
> Or, is it possible to run more than one node per machine for testing?
> 
> I need to be able to run 100+ indices with 10+ mappings per index, all  
> fairly low use. One machine should be able to handle the load for  
> testing.
> 
> ...Ken
> 
> Shay Banon wrote:
> 
> > Have you configured a gateway for elasticsearch? The configuration is  
> > simple:
> > 
> > gateway:  
> > type: fs  
> > fs:  
> > location: /path/to/shared/location
> > 
> > -shay.banon
> > 
> > On Thu, Jul 8, 2010 at 2:16 AM, Kenneth Loafman  
> > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > 
> > ```
> > Hi,
> > 
> > I installed elasticsearch-0.8.0 on 64-bit Ubuntu 9.10 system. It
> > 
> > ```
> 
> comes
> 
> > ```
> > up, runs fine, creates the indices I want, etc., but has a very
> > 
> > ```
> 
> serious
> 
> > ```
> > problem I can't trace down. If I do a 'bin/service/elasticsearch
> > restart' command (change config/restart), the entire database
> > disappears. I'm fairly certain this is not the desired action.
> > 
> > ```
> 
> Here's
> 
> > ```
> > a console log showing what happens:
> > 
> > ken@stealth:/opt/elasticsearch-0.8.0$ ll work/elasticsearch/indices/
> > total 4
> > drwxr-xr-x 22 ken ken 4096 2010-07-07 17:48
> > 74f73a5f-50a0-4ad4-a574-6466300b9399
> > 
> > <go off and run ES for a while, then restart>
> > 
> > ken@stealth:/opt/elasticsearch-0.8.0$ bin/service/elasticsearch
> > 
> > ```
> 
> restart
> 
> > ```
> > Stopping ElasticSearch...
> > Stopped ElasticSearch.
> > Starting ElasticSearch...Waiting for ElasticSearch......
> > ElasticSearch started.
> > 
> > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/
> > ls: cannot access
> > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/: No
> > such file or directory
> > ken@stealth:/opt/elasticsearch-0.8.0$ ll work/elasticsearch/indices/
> > total 0
> > 
> > I've read through the docs many times (very terse), changed config,
> > blown it away and unzipped again, nothing helps. Perhaps you can
> > 
> > ```
> 
> tell
> 
> > ```
> > me what I can do to correct the problem. I'm pretty sure it should
> > persist the data.
> > 
> > ...Thanks,
> > ...Ken
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 12, 2010, 6:09pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/5 "2010-07-12T18:09:29Z")

</div>

One last thing, if those indices are for very low use, I guess the default 5  
shards is too much, maybe 1 shard with 1 replica is enough?

On Mon, Jul 12, 2010 at 9:08 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Actually, the 100 shards per node is just arbitrary limit, and to be  
> honest, not properly designed in terms of configuration (let me see what I  
> can do for 0.9 regarding that). Second, creating so many indices / shards is  
> going to be pretty heavy on a single node. In 0.9, that overhead is going to  
> be lower, but still... . And yes, you can certainly start more than one node  
> on a single machine.
> 
> -shay.banon
> 
> On Mon, Jul 12, 2010 at 8:40 PM, Kenneth Loafman \<  
> [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> > Thanks! One more learning hurdle down.
> > 
> > Is there a list somewhere of all the configuration items that I could  
> > adjust? I need to increase the number of indices that I can build on a  
> > node. I'm getting 500 errors every time I add a new index or mapping.
> > 
> > As I understand it, given the limit of 100 shards per node, and the  
> > default of 5 shards per index, each node can support only 20 indices.  
> > Am I missing something?
> > 
> > Or, is it possible to run more than one node per machine for testing?
> > 
> > I need to be able to run 100+ indices with 10+ mappings per index, all  
> > fairly low use. One machine should be able to handle the load for  
> > testing.
> > 
> > ...Ken
> > 
> > Shay Banon wrote:
> > 
> > > Have you configured a gateway for elasticsearch? The configuration is  
> > > simple:
> > > 
> > > gateway:  
> > > type: fs  
> > > fs:  
> > > location: /path/to/shared/location
> > > 
> > > -shay.banon
> > > 
> > > On Thu, Jul 8, 2010 at 2:16 AM, Kenneth Loafman  
> > > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > > 
> > > ```
> > > Hi,
> > > 
> > > I installed elasticsearch-0.8.0 on 64-bit Ubuntu 9.10 system. It
> > > 
> > > ```
> > 
> > comes
> > 
> > > ```
> > > up, runs fine, creates the indices I want, etc., but has a very
> > > 
> > > ```
> > 
> > serious
> > 
> > > ```
> > > problem I can't trace down. If I do a 'bin/service/elasticsearch
> > > restart' command (change config/restart), the entire database
> > > disappears. I'm fairly certain this is not the desired action.
> > > 
> > > ```
> > 
> > Here's
> > 
> > > ```
> > > a console log showing what happens:
> > > 
> > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > > 
> > > ```
> > 
> > work/elasticsearch/indices/
> > 
> > > ```
> > > total 4
> > > drwxr-xr-x 22 ken ken 4096 2010-07-07 17:48
> > > 74f73a5f-50a0-4ad4-a574-6466300b9399
> > > 
> > > <go off and run ES for a while, then restart>
> > > 
> > > ken@stealth:/opt/elasticsearch-0.8.0$ bin/service/elasticsearch
> > > 
> > > ```
> > 
> > restart
> > 
> > > ```
> > > Stopping ElasticSearch...
> > > Stopped ElasticSearch.
> > > Starting ElasticSearch...Waiting for ElasticSearch......
> > > ElasticSearch started.
> > > 
> > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/
> > > ls: cannot access
> > > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/: No
> > > such file or directory
> > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > > 
> > > ```
> > 
> > work/elasticsearch/indices/
> > 
> > > ```
> > > total 0
> > > 
> > > I've read through the docs many times (very terse), changed config,
> > > blown it away and unzipped again, nothing helps. Perhaps you can
> > > 
> > > ```
> > 
> > tell
> > 
> > > ```
> > > me what I can do to correct the problem. I'm pretty sure it should
> > > persist the data.
> > > 
> > > ...Thanks,
> > > ...Ken
> > > 
> > > ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [July 12, 2010, 6:24pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/6 "2010-07-12T18:24:29Z")

</div>

I have to do that to keep the term frequencies unique per index. If  
there is a cleaner way to do it, I'd sure like to find out.

...Thanks,  
...Ken

Shay Banon wrote:

> Actually, the 100 shards per node is just arbitrary limit, and to be  
> honest, not properly designed in terms of configuration (let me see what  
> I can do for 0.9 regarding that). Second, creating so many indices /  
> shards is going to be pretty heavy on a single node. In 0.9, that  
> overhead is going to be lower, but still... . And yes, you can certainly  
> start more than one node on a single machine.
> 
> -shay.banon
> 
> On Mon, Jul 12, 2010 at 8:40 PM, Kenneth Loafman  
> \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> ```
> Thanks! One more learning hurdle down.
> 
> Is there a list somewhere of all the configuration items that I could
> adjust? I need to increase the number of indices that I can build on a
> node. I'm getting 500 errors every time I add a new index or mapping.
> 
> As I understand it, given the limit of 100 shards per node, and the
> default of 5 shards per index, each node can support only 20 indices.
> Am I missing something?
> 
> Or, is it possible to run more than one node per machine for testing?
> 
> I need to be able to run 100+ indices with 10+ mappings per index, all
> fairly low use. One machine should be able to handle the load for
> testing.
> 
> ...Ken
> 
> Shay Banon wrote:
> > Have you configured a gateway for elasticsearch? The configuration is
> > simple:
> >
> > gateway:
> > type: fs
> > fs:
> > location: /path/to/shared/location
> >
> > -shay.banon
> >
> > On Thu, Jul 8, 2010 at 2:16 AM, Kenneth Loafman
> > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>> wrote:
> >
> > Hi,
> >
> > I installed elasticsearch-0.8.0 on 64-bit Ubuntu 9.10 system.
> It comes
> > up, runs fine, creates the indices I want, etc., but has a
> very serious
> > problem I can't trace down. If I do a 'bin/service/elasticsearch
> > restart' command (change config/restart), the entire database
> > disappears. I'm fairly certain this is not the desired
> action. Here's
> > a console log showing what happens:
> >
> > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> work/elasticsearch/indices/
> > total 4
> > drwxr-xr-x 22 ken ken 4096 2010-07-07 17:48
> > 74f73a5f-50a0-4ad4-a574-6466300b9399
> >
> > <go off and run ES for a while, then restart>
> >
> > ken@stealth:/opt/elasticsearch-0.8.0$
> bin/service/elasticsearch restart
> > Stopping ElasticSearch...
> > Stopped ElasticSearch.
> > Starting ElasticSearch...Waiting for ElasticSearch......
> > ElasticSearch started.
> >
> > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/
> > ls: cannot access
> >    
> work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/: No
> > such file or directory
> > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> work/elasticsearch/indices/
> > total 0
> >
> > I've read through the docs many times (very terse), changed
> config,
> > blown it away and unzipped again, nothing helps. Perhaps you
> can tell
> > me what I can do to correct the problem. I'm pretty sure it
> should
> > persist the data.
> >
> > ...Thanks,
> > ...Ken
> >
> >
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 12, 2010, 6:56pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/7 "2010-07-12T18:56:23Z")

</div>

You can use smart filters on queries to execute to filter out what you want.  
For example, you can have an index per user, but you can also have a single  
index with a user field, and execute queries which filters based on the user  
field.

-shay.banon

On Mon, Jul 12, 2010 at 9:24 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> I have to do that to keep the term frequencies unique per index. If  
> there is a cleaner way to do it, I'd sure like to find out.
> 
> ...Thanks,  
> ...Ken
> 
> Shay Banon wrote:
> 
> > Actually, the 100 shards per node is just arbitrary limit, and to be  
> > honest, not properly designed in terms of configuration (let me see what  
> > I can do for 0.9 regarding that). Second, creating so many indices /  
> > shards is going to be pretty heavy on a single node. In 0.9, that  
> > overhead is going to be lower, but still... . And yes, you can certainly  
> > start more than one node on a single machine.
> > 
> > -shay.banon
> > 
> > On Mon, Jul 12, 2010 at 8:40 PM, Kenneth Loafman  
> > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > 
> > ```
> > Thanks! One more learning hurdle down.
> > 
> > Is there a list somewhere of all the configuration items that I could
> > adjust? I need to increase the number of indices that I can build on
> > 
> > ```
> 
> a
> 
> > ```
> > node. I'm getting 500 errors every time I add a new index or
> > 
> > ```
> 
> mapping.
> 
> > ```
> > As I understand it, given the limit of 100 shards per node, and the
> > default of 5 shards per index, each node can support only 20 indices.
> > Am I missing something?
> > 
> > Or, is it possible to run more than one node per machine for testing?
> > 
> > I need to be able to run 100+ indices with 10+ mappings per index,
> > 
> > ```
> 
> all
> 
> > ```
> > fairly low use. One machine should be able to handle the load for
> > testing.
> > 
> > ...Ken
> > 
> > Shay Banon wrote:
> > > Have you configured a gateway for elasticsearch? The configuration
> > 
> > ```
> 
> is
> 
> > ```
> > > simple:
> > >
> > > gateway:
> > > type: fs
> > > fs:
> > > location: /path/to/shared/location
> > >
> > > -shay.banon
> > >
> > > On Thu, Jul 8, 2010 at 2:16 AM, Kenneth Loafman
> > > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > >
> > > Hi,
> > >
> > > I installed elasticsearch-0.8.0 on 64-bit Ubuntu 9.10 system.
> > It comes
> > > up, runs fine, creates the indices I want, etc., but has a
> > very serious
> > > problem I can't trace down. If I do a
> > 
> > ```
> 
> 'bin/service/elasticsearch
> 
> > ```
> > > restart' command (change config/restart), the entire database
> > > disappears. I'm fairly certain this is not the desired
> > action. Here's
> > > a console log showing what happens:
> > >
> > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > work/elasticsearch/indices/
> > > total 4
> > > drwxr-xr-x 22 ken ken 4096 2010-07-07 17:48
> > > 74f73a5f-50a0-4ad4-a574-6466300b9399
> > >
> > > <go off and run ES for a while, then restart>
> > >
> > > ken@stealth:/opt/elasticsearch-0.8.0$
> > bin/service/elasticsearch restart
> > > Stopping ElasticSearch...
> > > Stopped ElasticSearch.
> > > Starting ElasticSearch...Waiting for ElasticSearch......
> > > ElasticSearch started.
> > >
> > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > >
> > 
> > ```
> 
> work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/
> 
> > ```
> > > ls: cannot access
> > >
> > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/: No
> > > such file or directory
> > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > work/elasticsearch/indices/
> > > total 0
> > >
> > > I've read through the docs many times (very terse), changed
> > config,
> > > blown it away and unzipped again, nothing helps. Perhaps you
> > can tell
> > > me what I can do to correct the problem. I'm pretty sure it
> > should
> > > persist the data.
> > >
> > > ...Thanks,
> > > ...Ken
> > >
> > >
> > >
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\
**Post date:** [July 12, 2010, 7:29pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/8 "2010-07-12T19:29:20Z")

</div>

Shay,

Can you shed some more light into the inner workings, the potential overhead  
due to the high number of indices?

There seems to be some pros to using many indices but it sounds like the  
benefits need to be balanced with the associated overhead.  
For example, we're planning to shard (log data) by date. This means when  
index is updated, it's not updating a very large index, and indices for  
older days can be better cached. However this would mean hundreds of indices  
per log (as time goes), hence I'm trying to understand whether we need to  
consider sharding by week/month instead, etc.

Regards,  
Berkay Mollamustafaoglu  
mberkay on yahoo, google and skype

On Mon, Jul 12, 2010 at 2:08 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Actually, the 100 shards per node is just arbitrary limit, and to be  
> honest, not properly designed in terms of configuration (let me see what I  
> can do for 0.9 regarding that). Second, creating so many indices / shards is  
> going to be pretty heavy on a single node. In 0.9, that overhead is going to  
> be lower, but still... . And yes, you can certainly start more than one node  
> on a single machine.
> 
> -shay.banon
> 
> On Mon, Jul 12, 2010 at 8:40 PM, Kenneth Loafman \<  
> [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> > Thanks! One more learning hurdle down.
> > 
> > Is there a list somewhere of all the configuration items that I could  
> > adjust? I need to increase the number of indices that I can build on a  
> > node. I'm getting 500 errors every time I add a new index or mapping.
> > 
> > As I understand it, given the limit of 100 shards per node, and the  
> > default of 5 shards per index, each node can support only 20 indices.  
> > Am I missing something?
> > 
> > Or, is it possible to run more than one node per machine for testing?
> > 
> > I need to be able to run 100+ indices with 10+ mappings per index, all  
> > fairly low use. One machine should be able to handle the load for  
> > testing.
> > 
> > ...Ken
> > 
> > Shay Banon wrote:
> > 
> > > Have you configured a gateway for elasticsearch? The configuration is  
> > > simple:
> > > 
> > > gateway:  
> > > type: fs  
> > > fs:  
> > > location: /path/to/shared/location
> > > 
> > > -shay.banon
> > > 
> > > On Thu, Jul 8, 2010 at 2:16 AM, Kenneth Loafman  
> > > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > > 
> > > ```
> > > Hi,
> > > 
> > > I installed elasticsearch-0.8.0 on 64-bit Ubuntu 9.10 system. It
> > > 
> > > ```
> > 
> > comes
> > 
> > > ```
> > > up, runs fine, creates the indices I want, etc., but has a very
> > > 
> > > ```
> > 
> > serious
> > 
> > > ```
> > > problem I can't trace down. If I do a 'bin/service/elasticsearch
> > > restart' command (change config/restart), the entire database
> > > disappears. I'm fairly certain this is not the desired action.
> > > 
> > > ```
> > 
> > Here's
> > 
> > > ```
> > > a console log showing what happens:
> > > 
> > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > > 
> > > ```
> > 
> > work/elasticsearch/indices/
> > 
> > > ```
> > > total 4
> > > drwxr-xr-x 22 ken ken 4096 2010-07-07 17:48
> > > 74f73a5f-50a0-4ad4-a574-6466300b9399
> > > 
> > > <go off and run ES for a while, then restart>
> > > 
> > > ken@stealth:/opt/elasticsearch-0.8.0$ bin/service/elasticsearch
> > > 
> > > ```
> > 
> > restart
> > 
> > > ```
> > > Stopping ElasticSearch...
> > > Stopped ElasticSearch.
> > > Starting ElasticSearch...Waiting for ElasticSearch......
> > > ElasticSearch started.
> > > 
> > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/
> > > ls: cannot access
> > > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/: No
> > > such file or directory
> > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > > 
> > > ```
> > 
> > work/elasticsearch/indices/
> > 
> > > ```
> > > total 0
> > > 
> > > I've read through the docs many times (very terse), changed config,
> > > blown it away and unzipped again, nothing helps. Perhaps you can
> > > 
> > > ```
> > 
> > tell
> > 
> > > ```
> > > me what I can do to correct the problem. I'm pretty sure it should
> > > persist the data.
> > > 
> > > ...Thanks,
> > > ...Ken
> > > 
> > > ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 12, 2010, 7:49pm UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/9 "2010-07-12T19:49:01Z")

</div>

Its really up to the number of machines you plan to have and how the load  
(in this case, shards) will be distributed among them. Each shard is a  
Lucene index, which does come with its overhead (though I reduced it in  
0.9). For time based indexing, it make sense to use different indices (which  
give you another dimension to scaling out, since you can't change the number  
of shards per index).

One option that you can implement is to do (for example) week based index  
files for the past 4 weeks, and then, do 1 month based indexing for older  
data. This will require reindexing a week that has passed the one month  
marker into a new index that represents a month.

I would say hundreds of indices per log is a bit much on a small cluster of  
nodes (\< 5), so either increase the time span, use filters, or do something  
similar to what I suggested above. In any case, the amount of data per index  
you can store in ES is quite large (but requires testing on your end to see  
it fits what you try to do).

-shay.banon

On Mon, Jul 12, 2010 at 10:29 PM, Berkay Mollamustafaoglu \<[mberkay@gmail.com](mailto:mberkay@gmail.com)

> wrote:

> Shay,
> 
> Can you shed some more light into the inner workings, the potential  
> overhead due to the high number of indices?
> 
> There seems to be some pros to using many indices but it sounds like the  
> benefits need to be balanced with the associated overhead.  
> For example, we're planning to shard (log data) by date. This means when  
> index is updated, it's not updating a very large index, and indices for  
> older days can be better cached. However this would mean hundreds of indices  
> per log (as time goes), hence I'm trying to understand whether we need to  
> consider sharding by week/month instead, etc.
> 
> Regards,  
> Berkay Mollamustafaoglu  
> mberkay on yahoo, google and skype
> 
> On Mon, Jul 12, 2010 at 2:08 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > Actually, the 100 shards per node is just arbitrary limit, and to be  
> > honest, not properly designed in terms of configuration (let me see what I  
> > can do for 0.9 regarding that). Second, creating so many indices / shards is  
> > going to be pretty heavy on a single node. In 0.9, that overhead is going to  
> > be lower, but still... . And yes, you can certainly start more than one node  
> > on a single machine.
> > 
> > -shay.banon
> > 
> > On Mon, Jul 12, 2010 at 8:40 PM, Kenneth Loafman \<  
> > [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > 
> > > Thanks! One more learning hurdle down.
> > > 
> > > Is there a list somewhere of all the configuration items that I could  
> > > adjust? I need to increase the number of indices that I can build on a  
> > > node. I'm getting 500 errors every time I add a new index or mapping.
> > > 
> > > As I understand it, given the limit of 100 shards per node, and the  
> > > default of 5 shards per index, each node can support only 20 indices.  
> > > Am I missing something?
> > > 
> > > Or, is it possible to run more than one node per machine for testing?
> > > 
> > > I need to be able to run 100+ indices with 10+ mappings per index, all  
> > > fairly low use. One machine should be able to handle the load for  
> > > testing.
> > > 
> > > ...Ken
> > > 
> > > Shay Banon wrote:
> > > 
> > > > Have you configured a gateway for elasticsearch? The configuration is  
> > > > simple:
> > > > 
> > > > gateway:  
> > > > type: fs  
> > > > fs:  
> > > > location: /path/to/shared/location
> > > > 
> > > > -shay.banon
> > > > 
> > > > On Thu, Jul 8, 2010 at 2:16 AM, Kenneth Loafman  
> > > > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > > > 
> > > > ```
> > > > Hi,
> > > > 
> > > > I installed elasticsearch-0.8.0 on 64-bit Ubuntu 9.10 system. It
> > > > 
> > > > ```
> > > 
> > > comes
> > > 
> > > > ```
> > > > up, runs fine, creates the indices I want, etc., but has a very
> > > > 
> > > > ```
> > > 
> > > serious
> > > 
> > > > ```
> > > > problem I can't trace down. If I do a 'bin/service/elasticsearch
> > > > restart' command (change config/restart), the entire database
> > > > disappears. I'm fairly certain this is not the desired action.
> > > > 
> > > > ```
> > > 
> > > Here's
> > > 
> > > > ```
> > > > a console log showing what happens:
> > > > 
> > > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > > > 
> > > > ```
> > > 
> > > work/elasticsearch/indices/
> > > 
> > > > ```
> > > > total 4
> > > > drwxr-xr-x 22 ken ken 4096 2010-07-07 17:48
> > > > 74f73a5f-50a0-4ad4-a574-6466300b9399
> > > > 
> > > > <go off and run ES for a while, then restart>
> > > > 
> > > > ken@stealth:/opt/elasticsearch-0.8.0$ bin/service/elasticsearch
> > > > 
> > > > ```
> > > 
> > > restart
> > > 
> > > > ```
> > > > Stopping ElasticSearch...
> > > > Stopped ElasticSearch.
> > > > Starting ElasticSearch...Waiting for ElasticSearch......
> > > > ElasticSearch started.
> > > > 
> > > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > > > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/
> > > > ls: cannot access
> > > > work/elasticsearch/indices/74f73a5f-50a0-4ad4-a574-6466300b9399/:
> > > > 
> > > > ```
> > > 
> > > No
> > > 
> > > > ```
> > > > such file or directory
> > > > ken@stealth:/opt/elasticsearch-0.8.0$ ll
> > > > 
> > > > ```
> > > 
> > > work/elasticsearch/indices/
> > > 
> > > > ```
> > > > total 0
> > > > 
> > > > I've read through the docs many times (very terse), changed config,
> > > > blown it away and unzipped again, nothing helps. Perhaps you can
> > > > 
> > > > ```
> > > 
> > > tell
> > > 
> > > > ```
> > > > me what I can do to correct the problem. I'm pretty sure it should
> > > > persist the data.
> > > > 
> > > > ...Thanks,
> > > > ...Ken
> > > > 
> > > > ```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:22am UTC](https://discuss.elastic.co/t/getting-data-to-persist/3078/10 "2017-07-06T04:22:25Z")

</div>