Hi,
We're trying to build an index of 2 Billion documents and have been
experiencing some issues with elasticsearch. Here's the summary.
In order to make tests and run queries against, I've deployed a single node
elasticsearch instance with 1.4 billion documents (typical microblog
content) in 1 index with 5 shards using 1.2TB disk space. For all search
requests, I've got response in approximately 5-6 secs, even got facet
results with date range filters. Any other facet queries esspecially terms
facets would cause OOM Exception. That's something I'd expect as faceting
all happens in memory. Although the instance had 30 GB heap and used 10-12
GB actively, such facet queries failed.
In the second iteration we decided to split documents to 10 indexes with 10
shards. 150-200GB for every index. However We immediately run out of heap
space and reduced the number of shards to 5. Another change was our
document featured a parent-child relationship. 2 billion documents would
consist of 10-30 million parents with rest being children. The index for
both were chosen based on a timestamp, therefore almost all indexes had the
same parents. That meant multiple index queries would yield same parent
more than once in the results, which was a complication we'd be considering
in our application.
In this setup we were only able to keep 5 indexes (of 5 shards) online with
a 80 GB heap. During indexing we had to close at least 2 indexes and index
against only 3 indexes to keep heap level at 70-80%. Anyhow, we were able
to complete indexing. But the initial startup of the node would take 10s of
minutes and use 20-50GB of heap. Even leaving 48 GB to OS would not help.
At this point, we had 2 nodes with 5 indexes of 5 shards in each nodes.
We've decided to join them into a cluster. Since we were still testing,
we've set number of replicas to zero, set the cluster names and ip settings
and waited. After 12 hours and continuously observed IO, one of the nodes
choked and came to halt. After closing both nodes as they were unresponsive
and trying to open again the most unexpected and horrific thing happened.
Some shards got lost. This is a horrible thing to experience but they were
gone for good. Even the directory were not there. Simple vaporized. Having
backed up lucene directories, we were able to recover shard by shard but it
was a very bad experience.
Having explained the scenario, here are a couple of questions.
1- With 30 GB of heap, I'm only able to keep 4 indexes of 200GB size online
with parent/child mapping. If I didn't have parent/child documents would my
index use much less heap? Is parent/child relationship a big overhead.
2- Why are my indexes using that amount of memory. Most of the fields are
not analyzed (tags, user attributes and names). Does it create an
additional overhead as cardinality of unique field values across the index
would be high as opposed to analyzed fields.
3- I'm seeing that elastic is being used in AWS EC2. With my heap usage I
find it hard to deploy to cloud, and I feel that I'm definitely missing
something as 80 GB heap is not a common practice in EC2 deployments. For 2B
documents of ~3TB with parent child relationship, how many indices and
shard should I have? Any recommendations on index partitioning and sizing?
4- How come it's possible to get a shard lost? How can that happen?
Thanks in advance,
Ukyo
PS: Here's my elastic.yml
for fast indexing
#indices.memory.index_buffer_size: 10%
#indices.memory.min_shard_index_buffer_size: 12mb
#indices.memory.min_index_buffer_size: 200mb
#index.translog.flush_threshold_ops: 50000
Bulk pool
threadpool.bulk.type: fixed
threadpool.bulk.size: 60
threadpool.bulk.queue_size: 300
threadpool.index.type: fixed
threadpool.index.size: 60
threadpool.index.queue_size: 200
threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: 100
cluster.routing.allocation.node_initial_primaries_recoveries: 4
cluster.routing.allocation.node_concurrent_recoveries: 15
indices.recovery.max_bytes_per_sec: 100mb
indices.recovery.concurrent_streams: 5
cluster.routing.allocation.disable_allocation: true
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5b376573-6d8a-4fd8-95ab-9d98010bacbf%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.