Suggestion needed on Indexing Performance

Hi,

Here is our ES ( elasticsearch-1.2.1 ) cluster details

  • 20 Data Nodes (physical machines 2TB normal disk,32GB RAM,8 cpu
    cores). Both Xms & Xmx 14GB. Xmn 7GB
  • 2 Master Nodes (virtual machines 16GB RAM)
  • Cluster Size ~17 TB (out of 20 * 2 = 40TB)
  • Number of records ~70 billions
  • Number of indices ~3500
  • Number of shards ~10000

Indices config

  • index.refresh_interval: 40s
  • index.compound_format: true
  • index.compound_on_flush: true
  • index.merge.policy.max_merged_segment: 500mb
  • index.codec.postings_format.no_bloom.type: default
  • indices.cache.filter.size: 1000mb
  • indices.store.throttle.type: merge
  • indices.store.throttle.max_bytes_per_sec: 30mb
  • index.routing.allocation.total_shards_per_node: 2

We create 100 daily indices. We keep indices maximum of 30 days.

  • Small size indices (1GB to 5GB / per day - one shard & one replica) .
  • Large size indices (95GB to 125GB / per day - 12 shards & one replica)
  • We issue bulk indexing request . 1000 docs / per request.
  • docs may be simple tomcat logs / exceptions

100000 - 150000 search queries per day.

  • Queries may be simple term query or phrase query.
  • 80% of the queries response time fall under 10s. (Which is acceptable).
  • Queries on large indices (12 shards) , response time fall under 20s to
    30s

Problem (Suggestions needed)

  • indexing time for bulk request(1000 per req) for first 5 to 8 requests
    is less than 500 ms .
  • Then next 5 to 8 indexing bulk requests(1000 per req), response time
    is 2000ms to 5000ms (Some times 20000 ms or above)
  • Then next 5 to 8 indexing bulk requests(1000 per req), response time
    is 1000ms to 2000ms
  • Then next 5 to 8 indexing bulk requests(1000 per req), response time
    is less than 500ms

This variation in response time continues as cycle only in large indices
(12 shards).

As a result it takes nearly 10 to 15 mins to index a file which contains
~400000 (4 lakh) docs on large indices (12 shards)

I don't change any settings related with Lucene Merge .

Over the time indexing process lagging far behind, then i update
index.number_of_replicas as 0 to large indices. (But we need
HighAvailability.)

  • What makes indexing slow ? ( Lucene Merge ? ES flush ? ES refresh ?
    ) How to identify ?
  • 12 shards - too big ?
  • I dont want to change index.number_of_replicas as 0 each time.
    Suggestions ?
  • Some times even i reduced the copy, indexing not getting catch up.
  • 10% of memory for indexing 1400mb for ~20 shards on each node for
    active indices (Obviously today indices.)
  • Therefore 70mb for each index. Whether i need to increase ?

FYI : I went through Mike's recent blog on performance for ES indexing.

Regards,
Anantha Govindarajan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d49106f0-3169-4bf6-83f4-c2782d51ab4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.