$es_heap_size

Hi Guys,

If i Set $ES_HEAP_SIZE to 64g should i see that the system is reserving
64GB of memory using htop?

Regards
Shawn

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote:

Hi Guys,

If i Set $ES_HEAP_SIZE to 64g should i see that the system is
reserving 64GB of memory using htop?

you should. note that using more than 30GB for the heap will prevent
the jvm from using compressed pointers, and will increase GC load. you
probably don't want to do this

clint

Regards
Shawn

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

What is i set it exactly to 30GB do i still get this problem.

On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote:

Hi Guys,

If i Set $ES_HEAP_SIZE to 64g should i see that the system is
reserving 64GB of memory using htop?

you should. note that using more than 30GB for the heap will prevent
the jvm from using compressed pointers, and will increase GC load. you
probably don't want to do this

clint

Regards
Shawn

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote:

What is i set it exactly to 30GB do i still get this problem.

?

On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote:
> Hi Guys,
>
>
> If i Set $ES_HEAP_SIZE to 64g should i see that the system
is
> reserving 64GB of memory using htop?

    you should.  note that using more than 30GB for the heap will
    prevent 
    the jvm from using compressed pointers, and will increase GC
    load. you 
    probably don't want to do this 
    
    clint 
    
    > 
    > 
    > Regards 
    > Shawn 
    > 
    > -- 
    > You received this message because you are subscribed to the
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit
    https://groups.google.com/groups/opt_out. 
    >   
    >   

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

If i set the ES_HEAP_SIZE to 30GB would i avoid the compressed pointers
problem?

On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote:

What is i set it exactly to 30GB do i still get this problem.

?

On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote:
> Hi Guys,
>
>
> If i Set $ES_HEAP_SIZE to 64g should i see that the system
is
> reserving 64GB of memory using htop?

    you should.  note that using more than 30GB for the heap will 
    prevent 
    the jvm from using compressed pointers, and will increase GC 
    load. you 
    probably don't want to do this 
    
    clint 
    
    > 
    > 
    > Regards 
    > Shawn 
    > 
    > -- 
    > You received this message because you are subscribed to the 
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails 
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit 
    https://groups.google.com/groups/opt_out. 
    >   
    >   

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote:

If i set the ES_HEAP_SIZE to 30GB would i avoid the compressed
pointers problem?

yes :slight_smile:

don't forget that you should leave about 50% of your RAM for kernel
filesystem cache as well

clint

On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote:
> What is i set it exactly to 30GB do i still get this
problem.

    ? 
    
    > 
    > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley
    wrote: 
    >         On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie
    wrote: 
    >         > Hi Guys, 
    >         > 
    >         > 
    >         > If i Set $ES_HEAP_SIZE to 64g should i see that
    the system 
    >         is 
    >         > reserving 64GB of memory using htop? 
    >         
    >         you should.  note that using more than 30GB for the
    heap will 
    >         prevent 
    >         the jvm from using compressed pointers, and will
    increase GC 
    >         load. you 
    >         probably don't want to do this 
    >         
    >         clint 
    >         
    >         > 
    >         > 
    >         > Regards 
    >         > Shawn 
    >         > 
    >         > -- 
    >         > You received this message because you are
    subscribed to the 
    >         Google 
    >         > Groups "elasticsearch" group. 
    >         > To unsubscribe from this group and stop receiving
    emails 
    >         from it, send 
    >         > an email to elasticsearc...@googlegroups.com. 
    >         > For more options, visit 
    >         https://groups.google.com/groups/opt_out. 
    >         >   
    >         >   
    >         
    >         
    > 
    > -- 
    > You received this message because you are subscribed to the
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit
    https://groups.google.com/groups/opt_out. 
    >   
    >   

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Shouldn't be problematic as the server has 128GB of RAM. So if i set that
ES_HEAP_SIZE to 30GB and run elastic search using top should i see elastic
search using up 30GB of ram?

Regards
Shawn

On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote:

If i set the ES_HEAP_SIZE to 30GB would i avoid the compressed
pointers problem?

yes :slight_smile:

don't forget that you should leave about 50% of your RAM for kernel
filesystem cache as well

clint

On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote:
> What is i set it exactly to 30GB do i still get this
problem.

    ? 
    
    > 
    > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley 
    wrote: 
    >         On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie 
    wrote: 
    >         > Hi Guys, 
    >         > 
    >         > 
    >         > If i Set $ES_HEAP_SIZE to 64g should i see that 
    the system 
    >         is 
    >         > reserving 64GB of memory using htop? 
    >         
    >         you should.  note that using more than 30GB for the 
    heap will 
    >         prevent 
    >         the jvm from using compressed pointers, and will 
    increase GC 
    >         load. you 
    >         probably don't want to do this 
    >         
    >         clint 
    >         
    >         > 
    >         > 
    >         > Regards 
    >         > Shawn 
    >         > 
    >         > -- 
    >         > You received this message because you are 
    subscribed to the 
    >         Google 
    >         > Groups "elasticsearch" group. 
    >         > To unsubscribe from this group and stop receiving 
    emails 
    >         from it, send 
    >         > an email to elasticsearc...@googlegroups.com. 
    >         > For more options, visit 
    >         https://groups.google.com/groups/opt_out. 
    >         >   
    >         >   
    >         
    >         
    > 
    > -- 
    > You received this message because you are subscribed to the 
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails 
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit 
    https://groups.google.com/groups/opt_out. 
    >   
    >   

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote:

Shouldn't be problematic as the server has 128GB of RAM. So if i set
that ES_HEAP_SIZE to 30GB and run elastic search using top should i
see elastic search using up 30GB of ram?

yes

with 128GB of RAM, you may consider running two instance of ES on the
same machine, but just make sure that you use "awareness" to ensure that
you don't have the same primary and replica shards on a single box.

http://www.elasticsearch.org/guide/reference/modules/cluster.html

clint

Regards
Shawn

On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote:
> If i set the ES_HEAP_SIZE to 30GB would i avoid the
compressed
> pointers problem?

    yes :) 
    
    don't forget that you should leave about 50% of your RAM for
    kernel 
    filesystem cache as well 
    
    clint 
    
    > 
    > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley
    wrote: 
    >         On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie
    wrote: 
    >         > What is i set it exactly to 30GB do i still get
    this 
    >         problem. 
    >         
    >         ? 
    >         
    >         > 
    >         > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton
    Gormley 
    >         wrote: 
    >         >         On Fri, 2013-02-08 at 01:08 -0800, Shawn
    Ritchie 
    >         wrote: 
    >         >         > Hi Guys, 
    >         >         > 
    >         >         > 
    >         >         > If i Set $ES_HEAP_SIZE to 64g should i
    see that 
    >         the system 
    >         >         is 
    >         >         > reserving 64GB of memory using htop? 
    >         >         
    >         >         you should.  note that using more than
    30GB for the 
    >         heap will 
    >         >         prevent 
    >         >         the jvm from using compressed pointers,
    and will 
    >         increase GC 
    >         >         load. you 
    >         >         probably don't want to do this 
    >         >         
    >         >         clint 
    >         >         
    >         >         > 
    >         >         > 
    >         >         > Regards 
    >         >         > Shawn 
    >         >         > 
    >         >         > -- 
    >         >         > You received this message because you
    are 
    >         subscribed to the 
    >         >         Google 
    >         >         > Groups "elasticsearch" group. 
    >         >         > To unsubscribe from this group and stop
    receiving 
    >         emails 
    >         >         from it, send 
    >         >         > an email to
    elasticsearc...@googlegroups.com. 
    >         >         > For more options, visit 
    >         >         https://groups.google.com/groups/opt_out. 
    >         >         >   
    >         >         >   
    >         >         
    >         >         
    >         > 
    >         > -- 
    >         > You received this message because you are
    subscribed to the 
    >         Google 
    >         > Groups "elasticsearch" group. 
    >         > To unsubscribe from this group and stop receiving
    emails 
    >         from it, send 
    >         > an email to elasticsearc...@googlegroups.com. 
    >         > For more options, visit 
    >         https://groups.google.com/groups/opt_out. 
    >         >   
    >         >   
    >         
    >         
    > 
    > -- 
    > You received this message because you are subscribed to the
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit
    https://groups.google.com/groups/opt_out. 
    >   
    >   

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Would I get greater throughput though using 2 instances?

Regards
Shawn

On Friday, 8 February 2013 12:11:24 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote:

Shouldn't be problematic as the server has 128GB of RAM. So if i set
that ES_HEAP_SIZE to 30GB and run elastic search using top should i
see elastic search using up 30GB of ram?

yes

with 128GB of RAM, you may consider running two instance of ES on the
same machine, but just make sure that you use "awareness" to ensure that
you don't have the same primary and replica shards on a single box.

http://www.elasticsearch.org/guide/reference/modules/cluster.html

clint

Regards
Shawn

On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote:
> If i set the ES_HEAP_SIZE to 30GB would i avoid the
compressed
> pointers problem?

    yes :) 
    
    don't forget that you should leave about 50% of your RAM for 
    kernel 
    filesystem cache as well 
    
    clint 
    
    > 
    > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley 
    wrote: 
    >         On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie 
    wrote: 
    >         > What is i set it exactly to 30GB do i still get 
    this 
    >         problem. 
    >         
    >         ? 
    >         
    >         > 
    >         > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton 
    Gormley 
    >         wrote: 
    >         >         On Fri, 2013-02-08 at 01:08 -0800, Shawn 
    Ritchie 
    >         wrote: 
    >         >         > Hi Guys, 
    >         >         > 
    >         >         > 
    >         >         > If i Set $ES_HEAP_SIZE to 64g should i 
    see that 
    >         the system 
    >         >         is 
    >         >         > reserving 64GB of memory using htop? 
    >         >         
    >         >         you should.  note that using more than 
    30GB for the 
    >         heap will 
    >         >         prevent 
    >         >         the jvm from using compressed pointers, 
    and will 
    >         increase GC 
    >         >         load. you 
    >         >         probably don't want to do this 
    >         >         
    >         >         clint 
    >         >         
    >         >         > 
    >         >         > 
    >         >         > Regards 
    >         >         > Shawn 
    >         >         > 
    >         >         > -- 
    >         >         > You received this message because you 
    are 
    >         subscribed to the 
    >         >         Google 
    >         >         > Groups "elasticsearch" group. 
    >         >         > To unsubscribe from this group and stop 
    receiving 
    >         emails 
    >         >         from it, send 
    >         >         > an email to 
    elasticsearc...@googlegroups.com. 
    >         >         > For more options, visit 
    >         >         https://groups.google.com/groups/opt_out. 
    >         >         >   
    >         >         >   
    >         >         
    >         >         
    >         > 
    >         > -- 
    >         > You received this message because you are 
    subscribed to the 
    >         Google 
    >         > Groups "elasticsearch" group. 
    >         > To unsubscribe from this group and stop receiving 
    emails 
    >         from it, send 
    >         > an email to elasticsearc...@googlegroups.com. 
    >         > For more options, visit 
    >         https://groups.google.com/groups/opt_out. 
    >         >   
    >         >   
    >         
    >         
    > 
    > -- 
    > You received this message because you are subscribed to the 
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails 
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit 
    https://groups.google.com/groups/opt_out. 
    >   
    >   

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-02-08 at 03:14 -0800, Shawn Ritchie wrote:

Would I get greater throughput though using 2 instances?

they'd be sharing CPU and IO but you'd be making use of all of your RAM
(as opposed to wasting half of it).

the answer is: I think so, but not entirely sure :slight_smile:

try it and see - let us know what you find

clint

Regards
Shawn

On Friday, 8 February 2013 12:11:24 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote:
> Shouldn't be problematic as the server has 128GB of RAM. So
if i set
> that ES_HEAP_SIZE to 30GB and run elastic search using top
should i
> see elastic search using up 30GB of ram?

    yes 
    
    with 128GB of RAM, you may consider running two instance of ES
    on the 
    same machine, but just make sure that you use "awareness" to
    ensure that 
    you don't have the same primary and replica shards on a single
    box. 
    
    http://www.elasticsearch.org/guide/reference/modules/cluster.html 
    
    clint 
    
    > 
    > 
    > Regards 
    > Shawn 
    > 
    > On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley
    wrote: 
    >         On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie
    wrote: 
    >         > If i set the ES_HEAP_SIZE to 30GB would i avoid
    the 
    >         compressed 
    >         > pointers problem? 
    >         
    >         yes :) 
    >         
    >         don't forget that you should leave about 50% of your
    RAM for 
    >         kernel 
    >         filesystem cache as well 
    >         
    >         clint 
    >         
    >         > 
    >         > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton
    Gormley 
    >         wrote: 
    >         >         On Fri, 2013-02-08 at 02:27 -0800, Shawn
    Ritchie 
    >         wrote: 
    >         >         > What is i set it exactly to 30GB do i
    still get 
    >         this 
    >         >         problem. 
    >         >         
    >         >         ? 
    >         >         
    >         >         > 
    >         >         > On Friday, 8 February 2013 11:22:36 UTC
    +1, Clinton 
    >         Gormley 
    >         >         wrote: 
    >         >         >         On Fri, 2013-02-08 at 01:08
    -0800, Shawn 
    >         Ritchie 
    >         >         wrote: 
    >         >         >         > Hi Guys, 
    >         >         >         > 
    >         >         >         > 
    >         >         >         > If i Set $ES_HEAP_SIZE to 64g
    should i 
    >         see that 
    >         >         the system 
    >         >         >         is 
    >         >         >         > reserving 64GB of memory using
    htop? 
    >         >         >         
    >         >         >         you should.  note that using
    more than 
    >         30GB for the 
    >         >         heap will 
    >         >         >         prevent 
    >         >         >         the jvm from using compressed
    pointers, 
    >         and will 
    >         >         increase GC 
    >         >         >         load. you 
    >         >         >         probably don't want to do this 
    >         >         >         
    >         >         >         clint 
    >         >         >         
    >         >         >         > 
    >         >         >         > 
    >         >         >         > Regards 
    >         >         >         > Shawn 
    >         >         >         > 
    >         >         >         > -- 
    >         >         >         > You received this message
    because you 
    >         are 
    >         >         subscribed to the 
    >         >         >         Google 
    >         >         >         > Groups "elasticsearch" group. 
    >         >         >         > To unsubscribe from this group
    and stop 
    >         receiving 
    >         >         emails 
    >         >         >         from it, send 
    >         >         >         > an email to 
    >         elasticsearc...@googlegroups.com. 
    >         >         >         > For more options, visit 
    >         >         >
    https://groups.google.com/groups/opt_out. 
    >         >         >         >   
    >         >         >         >   
    >         >         >         
    >         >         >         
    >         >         > 
    >         >         > -- 
    >         >         > You received this message because you
    are 
    >         subscribed to the 
    >         >         Google 
    >         >         > Groups "elasticsearch" group. 
    >         >         > To unsubscribe from this group and stop
    receiving 
    >         emails 
    >         >         from it, send 
    >         >         > an email to
    elasticsearc...@googlegroups.com. 
    >         >         > For more options, visit 
    >         >         https://groups.google.com/groups/opt_out. 
    >         >         >   
    >         >         >   
    >         >         
    >         >         
    >         > 
    >         > -- 
    >         > You received this message because you are
    subscribed to the 
    >         Google 
    >         > Groups "elasticsearch" group. 
    >         > To unsubscribe from this group and stop receiving
    emails 
    >         from it, send 
    >         > an email to elasticsearc...@googlegroups.com. 
    >         > For more options, visit 
    >         https://groups.google.com/groups/opt_out. 
    >         >   
    >         >   
    >         
    >         
    > 
    > -- 
    > You received this message because you are subscribed to the
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit
    https://groups.google.com/groups/opt_out. 
    >   
    >   

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hmm, i will need some help with this,

so what you are implying i run 2 instances of ElasticSearch with different
configs,

Set one instance with

node.rack_id: rack_one

cluster.routing.allocation.awareness.attributes: rack_id

node.master: true node.data: true
index.number_of_shards: 4
index.number_of_replicas: 0

set another instance with

node.rack_id: rack_two

cluster.routing.allocation.awareness.attributes: rack_two

node.master: false node.data: true
index.number_of_shards: 4
index.number_of_replicas: 0

and this will automatically 2 shards on each instance?

Regards
Shawn

On Friday, 8 February 2013 12:21:06 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 03:14 -0800, Shawn Ritchie wrote:

Would I get greater throughput though using 2 instances?

they'd be sharing CPU and IO but you'd be making use of all of your RAM
(as opposed to wasting half of it).

the answer is: I think so, but not entirely sure :slight_smile:

try it and see - let us know what you find

clint

Regards
Shawn

On Friday, 8 February 2013 12:11:24 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote:
> Shouldn't be problematic as the server has 128GB of RAM. So
if i set
> that ES_HEAP_SIZE to 30GB and run elastic search using top
should i
> see elastic search using up 30GB of ram?

    yes 
    
    with 128GB of RAM, you may consider running two instance of ES 
    on the 
    same machine, but just make sure that you use "awareness" to 
    ensure that 
    you don't have the same primary and replica shards on a single 
    box. 

http://www.elasticsearch.org/guide/reference/modules/cluster.html

    clint 
    
    > 
    > 
    > Regards 
    > Shawn 
    > 
    > On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley 
    wrote: 
    >         On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie 
    wrote: 
    >         > If i set the ES_HEAP_SIZE to 30GB would i avoid 
    the 
    >         compressed 
    >         > pointers problem? 
    >         
    >         yes :) 
    >         
    >         don't forget that you should leave about 50% of your 
    RAM for 
    >         kernel 
    >         filesystem cache as well 
    >         
    >         clint 
    >         
    >         > 
    >         > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton 
    Gormley 
    >         wrote: 
    >         >         On Fri, 2013-02-08 at 02:27 -0800, Shawn 
    Ritchie 
    >         wrote: 
    >         >         > What is i set it exactly to 30GB do i 
    still get 
    >         this 
    >         >         problem. 
    >         >         
    >         >         ? 
    >         >         
    >         >         > 
    >         >         > On Friday, 8 February 2013 11:22:36 UTC 
    +1, Clinton 
    >         Gormley 
    >         >         wrote: 
    >         >         >         On Fri, 2013-02-08 at 01:08 
    -0800, Shawn 
    >         Ritchie 
    >         >         wrote: 
    >         >         >         > Hi Guys, 
    >         >         >         > 
    >         >         >         > 
    >         >         >         > If i Set $ES_HEAP_SIZE to 64g 
    should i 
    >         see that 
    >         >         the system 
    >         >         >         is 
    >         >         >         > reserving 64GB of memory using 
    htop? 
    >         >         >         
    >         >         >         you should.  note that using 
    more than 
    >         30GB for the 
    >         >         heap will 
    >         >         >         prevent 
    >         >         >         the jvm from using compressed 
    pointers, 
    >         and will 
    >         >         increase GC 
    >         >         >         load. you 
    >         >         >         probably don't want to do this 
    >         >         >         
    >         >         >         clint 
    >         >         >         
    >         >         >         > 
    >         >         >         > 
    >         >         >         > Regards 
    >         >         >         > Shawn 
    >         >         >         > 
    >         >         >         > -- 
    >         >         >         > You received this message 
    because you 
    >         are 
    >         >         subscribed to the 
    >         >         >         Google 
    >         >         >         > Groups "elasticsearch" group. 
    >         >         >         > To unsubscribe from this group 
    and stop 
    >         receiving 
    >         >         emails 
    >         >         >         from it, send 
    >         >         >         > an email to 
    >         elasticsearc...@googlegroups.com. 
    >         >         >         > For more options, visit 
    >         >         > 
    https://groups.google.com/groups/opt_out. 
    >         >         >         >   
    >         >         >         >   
    >         >         >         
    >         >         >         
    >         >         > 
    >         >         > -- 
    >         >         > You received this message because you 
    are 
    >         subscribed to the 
    >         >         Google 
    >         >         > Groups "elasticsearch" group. 
    >         >         > To unsubscribe from this group and stop 
    receiving 
    >         emails 
    >         >         from it, send 
    >         >         > an email to 
    elasticsearc...@googlegroups.com. 
    >         >         > For more options, visit 
    >         >         https://groups.google.com/groups/opt_out. 
    >         >         >   
    >         >         >   
    >         >         
    >         >         
    >         > 
    >         > -- 
    >         > You received this message because you are 
    subscribed to the 
    >         Google 
    >         > Groups "elasticsearch" group. 
    >         > To unsubscribe from this group and stop receiving 
    emails 
    >         from it, send 
    >         > an email to elasticsearc...@googlegroups.com. 
    >         > For more options, visit 
    >         https://groups.google.com/groups/opt_out. 
    >         >   
    >         >   
    >         
    >         
    > 
    > -- 
    > You received this message because you are subscribed to the 
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails 
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit 
    https://groups.google.com/groups/opt_out. 
    >   
    >   

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-02-08 at 03:27 -0800, Shawn Ritchie wrote:

Hmm, i will need some help with this,

so what you are implying i run 2 instances of ElasticSearch with
different configs,

OK, first question is: do you have just one server or more? If you have
just one server then you are at risk. If that server dies, you will lose
your cluster and (eg if the disks die) lose your data.

With just one server, the "awareness" doesn't matter - if the server
dies, you're screwed.

If you have more than one server then Elasticsearch puts a primary shard
on one server and a replica of the primary on the other server. That
way, if you lose one server, all of your data is safe and the cluster
can keep on running.

If you are running two nodes on one server and two nodes on another
server, then you need to use awareness to ensure that the primary and
replica shards are not placed on the same server.

So the two nodes running on the same server would have the same rack ID.
That way Elasticsearch knows to place the primary shards on one rack,
and the replica shards on the other rack.

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

Basically we only have 1 machine for Elastic Search, and for now are
not interested in replication, for now we are only interested in maximising
performance.

So if I understood we do not need the awareness field! and the 2 elastic
search instances will take care of 2 shards each automatically?

Also each instance i'm guessing needs to be configured with different
ports? and which instance will i be querying?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Basically we only have 1 machine for Elastic Search, and for now are
not interested in replication, for now we are only interested in
maximising performance.

OK

So if I understood we do not need the awareness field! and the 2
elastic search instances will take care of 2 shards each
automatically?

Yes

Also each instance i'm guessing needs to be configured with different
ports? and which instance will i be querying?

Each node will try to connect to port 9300 and if that is not available,
try 9301, 9302 etc. This is the port for communication between nodes
(and for the java clients), so make sure that you have 9300 and 9301
open in your firewall.

Similarly, each node will bind to 9200, 9201.... for the HTTP
interface.

This is automatic - you don't need to configure anything except for
opening the ports on your firewall.

You can talk to either node (http on 9200 or 9201, java on 9300, 9301)

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

or do i only need to set node.max_local_storage_nodes: 2 and it will
automatically setup 2 instances?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-02-08 at 04:07 -0800, Shawn Ritchie wrote:

or do i only need to set node.max_local_storage_nodes: 2 and it will
automatically setup 2 instances?

No, you need to start two instances. the above setting just stops you
from starting more than X instances on the same server (or at least from
the same directory)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Last question :smiley:

How do i lunch and instance with a specific config file? As i find it quite
ironic that you specify the config file directory in the config file itself?

On Friday, 8 February 2013 13:09:10 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 04:07 -0800, Shawn Ritchie wrote:

or do i only need to set node.max_local_storage_nodes: 2 and it will
automatically setup 2 instances?

No, you need to start two instances. the above setting just stops you
from starting more than X instances on the same server (or at least from
the same directory)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-02-08 at 04:14 -0800, Shawn Ritchie wrote:

Last question :smiley:

How do i lunch and instance with a specific config file? As i find it
quite ironic that you specify the config file directory in the config
file itself?

http://www.elasticsearch.org/guide/reference/setup/configuration.html

On Friday, 8 February 2013 13:09:10 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 04:07 -0800, Shawn Ritchie wrote:
> or do i only need to set node.max_local_storage_nodes: 2 and
it will
> automatically setup 2 instances?

    No, you need to start two instances.  the above setting just
    stops you 
    from starting more than X instances on the same server (or at
    least from 
    the same directory) 

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Already read that post, but from what i understood or misunderstood, is its
making the assumption you will have 1 instance of elastic search running on
a machine.

What i'd like to do is with 1 elastic search installation is lunch 2
instance of elastic search with different /config /data /log node name.

Or is it that multiple versions of elastic search on the same machine run @
a directory level, that is 2 instances which are sharing the /config /data
and /log directries together with the node name?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-02-08 at 04:27 -0800, Shawn Ritchie wrote:

Already read that post, but from what i understood or misunderstood,
is its making the assumption you will have 1 instance of elastic
search running on a machine.

What i'd like to do is with 1 elastic search installation is lunch 2
instance of elastic search with different /config /data /log node
name.

Or is it that multiple versions of elastic search on the same machine
run @ a directory level, that is 2 instances which are sharing
the /config /data and /log directries together with the node name?

You can run multiple instances with the same paths (including logging
and data).

If you just want to specify a different node name, then you could do so
on the command line:

./bin/elasticsearch -Des.node.name=node_1
./bin/elasticsearch -Des.node.name=node_2

If you want to change more than that, you could specify a specific
config file:

./bin/elasticsearch -Des.config=/path/to/config/file_1
./bin/elasticsearch -Des.config=/path/to/config/file_2

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.