Elasticsearch client only cluster

Elastic Stack Elasticsearch
11

ok, just playing around with 28 nodes I set shards=28 and replicas=0.

I now have 28 shards all of my nodes now which is balanced.
This is definitely an alternate setup that will work but I will still to
try to find out how to keep logs local to the rack using the templates you
listed below.

Perhaps I don't need to keep logs per rack as this seems to balance out the
storage pretty well.
What mechanism does elasticsearch use to keep the data balanced across all
nodes?

28 shards all reported on all nodes.
logstash-2013.02.28 27 p STARTED 2972868 2.5gb 2753226040 node4
..
logstash-2013.02.28 0 p STARTED 2972863 2.5gb 2764377828 node19

On Wed, Feb 27, 2013 at 2:58 AM, Clinton Gormley clint@traveljury.comwrote:

On Wed, 2013-02-27 at 00:16 -0800, B wrote:

Great, I checked out the docs and can came up with following config to
enable rack awareness for each instance that would look something like
this.

node:
name: node1
rack_id: rack1
cluster:
name: elasticsearch
routing:
allocation:
awareness:
attributes: rack_id

Since logstash creates an index per day(logstash-2013.02.26) per node,
would I have to do anything special to make sure that an index and
it's shards are created for each node individually?

If you want 30 indices on 30 different nodes, then you need to create
each index with a different name, and set the allocation on each index
to tie it to a single node.

Elasticsearch Platform — Find real-time answers at scale | Elastic

For instance, you can use index templates to say: if the index name
matches "node_1_*" then set index.routing.allocation.include.rack_id to
"node_1"

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

Currently only 5 shards are allocated in my shard=5 rep=0 cluster
receiving logs for +1k servers under low load. The goal is to have all
30 nodes to have an index started and receiving logs for its rack to
handle high load.

es shards -v
index shard pri/rep state docs size
bytes node
logstash-2013.02.26 0 p STARTED 28031927 22.9gb
24677604861 n7
logstash-2013.02.26 1 p STARTED 26853297 22gb
23641399741 n18
logstash-2013.02.26 2 p STARTED 28035826 22.9gb
24686606451 n21
logstash-2013.02.26 3 p STARTED 28033599 22.9gb
24695469792 n24
logstash-2013.02.26 4 p STARTED 28037600 22.9gb
24687686161 n5

Any recommendations on how to configure elasticsearch rack awareness
and routing to handle this.

On Tuesday, February 26, 2013 10:40:36 AM UTC-8, Clinton Gormley
wrote:

    > If I use the distributed system, can I force data to stay
    local to the
    > rack?

    Yes. Look for rack awareness in the docs

    >
    >
    >
    > On Tuesday, February 26, 2013 1:41:46 AM UTC-8, Clinton
    Gormley wrote:
    >         Hiya
    >
    >         OK, the picture is slowly evolving :)
    >
    >         On Mon, 2013-02-25 at 16:23 -0800, beha...@gmail.com
    wrote:
    >         > So I have 30 racks at a colo and have 1 es
    instance per
    >         rack.
    >         > The 1 es instance per rack is used to index all
    logs for
    >         that rack
    >         > only.
    >         >
    >         > Is there anyway for each instance to join a 30
    node cluster
    >         as a
    >         > (client only) and not replicate or shard data
    between them?
    >
    >         A "client" in Elasticsearch terminology doesn't hold
    any
    >         data.  Hence
    >         part of the confusion.  I think what you're asking
    is: Can I
    >         have an
    >         index on a single node in the cluster?
    >
    >         The answer is yes: you can create 30 indices, and
    specify rack
    >         awareness
    >         for each index, so that each index sits in a single
    rack.
    >         >
    >         > The purpose of the client only cluster would allow
    me to
    >         search one
    >         > instance and have es query all members for data.
    >
    >         Yes, you can connect to any node in the cluster and
    query one
    >         or more
    >         indices.  It will forward queries to all relevant
    nodes.
    >
    >         Note: I don't recommend this setup.  Especially with
    30 nodes,
    >         the
    >         chances of one of them going down is pretty high.
     Hardware
    >         fails.  With
    >         your current setup (esp if you don't have any
    replicas) then
    >         you run a
    >         good chance of losing data.
    >
    >         Why not just use Elasticsearch as the distributed
    system that
    >         it is
    >         intended to be?
    >
    >         clint
    >
    >
    >
    > --
    > You received this message because you are subscribed to the
    Google
    > Groups "elasticsearch" group.
    > To unsubscribe from this group and stop receiving emails
    from it, send
    > an email to elasticsearc...@googlegroups.com.
    > For more options, visit
    https://groups.google.com/groups/opt_out.
    >
    >

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/1ZUNmPHD8wY/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.