$es_heap_size

Sorry Again, What if i wanted to run them as Services? I tried looking in
the /service/elasticsearch.conf and init.d but with no luck

On Friday, 8 February 2013 13:40:19 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 04:27 -0800, Shawn Ritchie wrote:

Already read that post, but from what i understood or misunderstood,
is its making the assumption you will have 1 instance of elastic
search running on a machine.

What i'd like to do is with 1 Elasticsearch installation is lunch 2
instance of Elasticsearch with different /config /data /log node
name.

Or is it that multiple versions of Elasticsearch on the same machine
run @ a directory level, that is 2 instances which are sharing
the /config /data and /log directries together with the node name?

You can run multiple instances with the same paths (including logging
and data).

If you just want to specify a different node name, then you could do so
on the command line:

./bin/elasticsearch -Des.node.name=node_1
./bin/elasticsearch -Des.node.name=node_2

If you want to change more than that, you could specify a specific
config file:

./bin/elasticsearch -Des.config=/path/to/config/file_1
./bin/elasticsearch -Des.config=/path/to/config/file_2

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Note, even if you run just one ES instance, on a 128gb, with 30gb allocated to ES, you potentially don't loose much unless you really need ES to have more memory. Depending on your index size, the file system cache will kick in and use whatever it can making searches faster.

You can also set index.store.type to mmapfs, which will use memory mapped files for the index files, boosting the speed of your search even further.

I would start with just 1 instance on that box, and see if you really need another one.

On Feb 8, 2013, at 3:21 PM, Shawn Ritchie xritchie@gmail.com wrote:

Sorry Again, What if i wanted to run them as Services? I tried looking in the /service/elasticsearch.conf and init.d but with no luck

On Friday, 8 February 2013 13:40:19 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 04:27 -0800, Shawn Ritchie wrote:

Already read that post, but from what i understood or misunderstood,
is its making the assumption you will have 1 instance of elastic
search running on a machine.

What i'd like to do is with 1 Elasticsearch installation is lunch 2
instance of Elasticsearch with different /config /data /log node
name.

Or is it that multiple versions of Elasticsearch on the same machine
run @ a directory level, that is 2 instances which are sharing
the /config /data and /log directries together with the node name?

You can run multiple instances with the same paths (including logging
and data).

If you just want to specify a different node name, then you could do so
on the command line:

./bin/elasticsearch -Des.node.name=node_1
./bin/elasticsearch -Des.node.name=node_2

If you want to change more than that, you could specify a specific
config file:

./bin/elasticsearch -Des.config=/path/to/config/file_1
./bin/elasticsearch -Des.config=/path/to/config/file_2

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

在 2013年2月13日星期三UTC+8上午6时43分40秒,kimchy写道:

Note, even if you run just one ES instance, on a 128gb, with 30gb
allocated to ES, you potentially don't loose much unless you really need ES
to have more memory. Depending on your index size, the file system cache
will kick in and use whatever it can making searches faster.

You can also set index.store.type to mmapfs, which will use memory mapped
files for the index files, boosting the speed of your search even further.

Now we already have about 2TB index in our Elasticsearch cluster, is it
still OK to change index.store.type to mmapfs ?

I would start with just 1 instance on that box, and see if you really need
another one.

On Feb 8, 2013, at 3:21 PM, Shawn Ritchie <xrit...@gmail.com <javascript:>>
wrote:

Sorry Again, What if i wanted to run them as Services? I tried looking in
the /service/elasticsearch.conf and init.d but with no luck

On Friday, 8 February 2013 13:40:19 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 04:27 -0800, Shawn Ritchie wrote:

Already read that post, but from what i understood or misunderstood,
is its making the assumption you will have 1 instance of elastic
search running on a machine.

What i'd like to do is with 1 Elasticsearch installation is lunch 2
instance of Elasticsearch with different /config /data /log node
name.

Or is it that multiple versions of Elasticsearch on the same machine
run @ a directory level, that is 2 instances which are sharing
the /config /data and /log directries together with the node name?

You can run multiple instances with the same paths (including logging
and data).

If you just want to specify a different node name, then you could do so
on the command line:

./bin/elasticsearch -Des.node.name http://des.node.name/=node_1
./bin/elasticsearch -Des.node.name http://des.node.name/=node_2

If you want to change more than that, you could specify a specific
config file:

./bin/elasticsearch -Des.config=/path/to/config/file_1
./bin/elasticsearch -Des.config=/path/to/config/file_2

clint

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2181407c-dfa9-4c07-9b30-3c0501ec7709%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

在 2013年2月8日星期五UTC+8下午7时11分24秒,Clinton Gormley写道:

On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote:

Shouldn't be problematic as the server has 128GB of RAM. So if i set
that ES_HEAP_SIZE to 30GB and run Elasticsearch using top should i
see Elasticsearch using up 30GB of ram?

yes

with 128GB of RAM, you may consider running two instance of ES on the
same machine, but just make sure that you use "awareness" to ensure that
you don't have the same primary and replica shards on a single box.

by set cluster.routing.allocation.same_shard.host to true ?

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

Regards
Shawn

On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote:
On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote:
> If i set the ES_HEAP_SIZE to 30GB would i avoid the
compressed
> pointers problem?

    yes :) 
    
    don't forget that you should leave about 50% of your RAM for 
    kernel 
    filesystem cache as well 
    
    clint 
    
    > 
    > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley 
    wrote: 
    >         On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie 
    wrote: 
    >         > What is i set it exactly to 30GB do i still get 
    this 
    >         problem. 
    >         
    >         ? 
    >         
    >         > 
    >         > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton 
    Gormley 
    >         wrote: 
    >         >         On Fri, 2013-02-08 at 01:08 -0800, Shawn 
    Ritchie 
    >         wrote: 
    >         >         > Hi Guys, 
    >         >         > 
    >         >         > 
    >         >         > If i Set $ES_HEAP_SIZE to 64g should i 
    see that 
    >         the system 
    >         >         is 
    >         >         > reserving 64GB of memory using htop? 
    >         >         
    >         >         you should.  note that using more than 
    30GB for the 
    >         heap will 
    >         >         prevent 
    >         >         the jvm from using compressed pointers, 
    and will 
    >         increase GC 
    >         >         load. you 
    >         >         probably don't want to do this 
    >         >         
    >         >         clint 
    >         >         
    >         >         > 
    >         >         > 
    >         >         > Regards 
    >         >         > Shawn 
    >         >         > 
    >         >         > -- 
    >         >         > You received this message because you 
    are 
    >         subscribed to the 
    >         >         Google 
    >         >         > Groups "elasticsearch" group. 
    >         >         > To unsubscribe from this group and stop 
    receiving 
    >         emails 
    >         >         from it, send 
    >         >         > an email to 
    elasticsearc...@googlegroups.com. 
    >         >         > For more options, visit 
    >         >         https://groups.google.com/groups/opt_out. 
    >         >         >   
    >         >         >   
    >         >         
    >         >         
    >         > 
    >         > -- 
    >         > You received this message because you are 
    subscribed to the 
    >         Google 
    >         > Groups "elasticsearch" group. 
    >         > To unsubscribe from this group and stop receiving 
    emails 
    >         from it, send 
    >         > an email to elasticsearc...@googlegroups.com. 
    >         > For more options, visit 
    >         https://groups.google.com/groups/opt_out. 
    >         >   
    >         >   
    >         
    >         
    > 
    > -- 
    > You received this message because you are subscribed to the 
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails 
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit 
    https://groups.google.com/groups/opt_out. 
    >   
    >   

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/69aeffae-fce5-4924-9eb0-ed5684352e20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

What about the memory settings on Client Node where there is no index?
Still 30GB? That would be either causing waste of RAM or limiting the
ability of aggregations, knowing that we can only use field_data cache for
aggregations, which may be 70% of the heap. Can we set HEAP larger on
Client Node?

On Tuesday, February 12, 2013 5:43:40 PM UTC-5, kimchy wrote:

Note, even if you run just one ES instance, on a 128gb, with 30gb
allocated to ES, you potentially don't loose much unless you really need ES
to have more memory. Depending on your index size, the file system cache
will kick in and use whatever it can making searches faster.

You can also set index.store.type to mmapfs, which will use memory mapped
files for the index files, boosting the speed of your search even further.

I would start with just 1 instance on that box, and see if you really need
another one.

On Feb 8, 2013, at 3:21 PM, Shawn Ritchie <xrit...@gmail.com <javascript:>>
wrote:

Sorry Again, What if i wanted to run them as Services? I tried looking in
the /service/elasticsearch.conf and init.d but with no luck

On Friday, 8 February 2013 13:40:19 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 04:27 -0800, Shawn Ritchie wrote:

Already read that post, but from what i understood or misunderstood,
is its making the assumption you will have 1 instance of elastic
search running on a machine.

What i'd like to do is with 1 Elasticsearch installation is lunch 2
instance of Elasticsearch with different /config /data /log node
name.

Or is it that multiple versions of Elasticsearch on the same machine
run @ a directory level, that is 2 instances which are sharing
the /config /data and /log directries together with the node name?

You can run multiple instances with the same paths (including logging
and data).

If you just want to specify a different node name, then you could do so
on the command line:

./bin/elasticsearch -Des.node.name http://des.node.name/=node_1
./bin/elasticsearch -Des.node.name http://des.node.name/=node_2

If you want to change more than that, you could specify a specific
config file:

./bin/elasticsearch -Des.config=/path/to/config/file_1
./bin/elasticsearch -Des.config=/path/to/config/file_2

clint

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5e261a3f-521c-493b-84bc-59f86743b6c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You probably dont want to start elasticsearch using a one liner like that.

What you need to do is to just create another init.d script, the difference between them should be that they point at different /etc/sysconfig/elasticsearch files, in here you should put different configs, dont set it in the /usr/share/elasticsearch.in.sh (i think its there default)

With this you can also spin up a new instance for master node, just put the differences in the sysconfig file, different heap etc etc

What you need to do is to just create another init.d script, the difference
between them should be that they point at different
/etc/sysconfig/elasticsearch files, in here you should put different
configs, dont set it in the /usr/share/elasticsearch.in.sh (i think its
there default)

With this you can also spin up a new instance for master node, just put the
differences in the sysconfig file, different heap etc etc

Den fredagen den 8:e februari 2013 kl. 15:21:39 UTC+1 skrev Shawn Ritchie:

Sorry Again, What if i wanted to run them as Services? I tried looking in
the /service/elasticsearch.conf and init.d but with no luck

On Friday, 8 February 2013 13:40:19 UTC+1, Clinton Gormley wrote:

On Fri, 2013-02-08 at 04:27 -0800, Shawn Ritchie wrote:

Already read that post, but from what i understood or misunderstood,
is its making the assumption you will have 1 instance of elastic
search running on a machine.

What i'd like to do is with 1 Elasticsearch installation is lunch 2
instance of Elasticsearch with different /config /data /log node
name.

Or is it that multiple versions of Elasticsearch on the same machine
run @ a directory level, that is 2 instances which are sharing
the /config /data and /log directries together with the node name?

You can run multiple instances with the same paths (including logging
and data).

If you just want to specify a different node name, then you could do so
on the command line:

./bin/elasticsearch -Des.node.name=node_1
./bin/elasticsearch -Des.node.name=node_2

If you want to change more than that, you could specify a specific
config file:

./bin/elasticsearch -Des.config=/path/to/config/file_1
./bin/elasticsearch -Des.config=/path/to/config/file_2

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e0192b3d-1ee3-4192-92e7-d4a3180a467d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.