# Completion Suggester indexing speed issues

**URL:** <https://discuss.elastic.co/t/completion-suggester-indexing-speed-issues/13336>\
**Category:** Elasticsearch\
**Created:** [August 26, 2013, 10:28pm UTC](https://discuss.elastic.co/t/completion-suggester-indexing-speed-issues/13336 "2013-08-26T22:28:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![robert\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_3/32/2132_2.png) [@robert\_3](https://discuss.elastic.co/u/robert_3)\
**Post date:** [August 26, 2013, 10:28pm UTC](https://discuss.elastic.co/t/completion-suggester-indexing-speed-issues/13336/1 "2013-08-26T22:28:28Z")

</div>

Hi all,

I've been testing out the new completion suggester in master, since it fits  
a use case I have pretty well. I've been having some consistent problems  
with slow indexing on my cluster, and have spent a while tweaking to try to  
remedy them but am now led to the conclusion that it's likely the  
completion suggester that's causing my slowdowns.

Here's my setup: I'm indexing approximately 2 billion docs, with two  
completion fields, each of which is has both input and output length of \<  
60 characters.  
We're using bulk indexing with various batch sizes with little difference  
in the end results. The index has 0 replicas during indexing, 10 shards,  
refresh\_interval = -1, warmers disabled. We saw heavy merge volume, so we  
increased segments\_per\_tier and index buffer size considerably. While we  
did get some speed improvements, we're still seeing a large number of  
queries in the indexing\_slowlog.log

The cluster is 3 master nodes, 10 data nodes, each of which are quad core  
and 16GB of memory. While running our bulk import scripts, we see the  
servers remaining at low CPU load and low disk io in iostat. We're using  
default garbage collector settings (+CondCardMark for Java 7) and we're not  
seeing large stop the world collections, and our heap usage sits well below  
the occupancy fraction. We're using the latest Java 7 release on Solaris 10  
(Joyent's SmartOS).

In the slowlog, we're seeing a large number docs that are taking anywhere  
from 500ms to 4s during indexing. A small sample from the slowlog  
(non-sequential):

[2013-08-26 22:17:03,626][TRACE][index.indexing.slowlog.index] [qup]  
[quizlet][8] took[549ms], took\_millis[549], type[words], id[2093528],  
routing[], source[{"id":2093528,"lang\_detect":"tension:en"}]  
[2013-08-26 22:17:39,694][TRACE][index.indexing.slowlog.index] [qup]  
[quizlet][8] took[526.3ms], took\_millis[526], type[words], id[2076947],  
routing[],  
source[{"id":2076947,"lang\_detect":"Puesto:es","lang\_completion":{"input":"es:en:Puesto:poner(to  
put/ to place)","output":"es:en:Puesto:poner(to put/ to place)"}}]  
[2013-08-26 22:19:43,308][TRACE][index.indexing.slowlog.index] [qup]  
[quizlet][8] took[784.7ms], took\_millis[784], type[words], id[2162549],  
routing[], source[{"id":2162549,"lang\_detect":"the  
teacher:en","lang\_completion":{"input":"en:es:the teacher:la  
maestra","output":"en:es:the teacher:la maestra"}}]

Our mapping for this type is as follows:

```
"words" : {
  "_all" : {
    "enabled" : false
  },
  "_source" : {
    "enabled" : false
  },
  "properties" : {
    "id" : {
      "type" : "long"
    },
    "lang_completion" : {
      "type" : "completion",
      "analyzer" : "foldedlowerkeyword",
      "payloads" : false,
      "preserve_separators" : true,
      "preserve_position_increments" : true
    },
    "lang_detect" : {
      "type" : "completion",
      "analyzer" : "foldedlowerkeyword",
      "payloads" : false,
      "preserve_separators" : true,
      "preserve_position_increments" : true
    }
  }

```

Analyzer to match that:

"index.analysis.analyzer.foldedlowerkeyword.filter.1" : "asciifolding",  
"index.analysis.analyzer.foldedlowerkeyword.filter.0" : "lowercase",  
"index.analysis.analyzer.foldedlowerkeyword.tokenizer" : "keyword",  
"index.analysis.analyzer.foldedlowerkeyword.type" : "custom",

Any tips on where to look further to diagnose this would be appreciated. At this point we've tweaked enough that I'm wondering about the performance profile of the completion suggestor, since it is a new feature and hasn't seen too much testing. Is there something we can provide to help dig into exactly where we are having slowdowns?

Thanks,

--

Robert Deaton

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\
**Post date:** [August 27, 2013, 10:37am UTC](https://discuss.elastic.co/t/completion-suggester-indexing-speed-issues/13336/2 "2013-08-27T10:37:08Z")

</div>

The only think I can think of here is that flushing takes a long time here  
and your index-concurrency is too low for the bulk requests.  
Maybe you can try to set:  
index.index\_concurrency = 16 (  
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings/))

The Completion stuff can be heavy on flush and merge so maybe hitting your  
cluster with too many requests will eventually slow it down. Can you try  
this setting and report back?

simon

On Tuesday, August 27, 2013 12:28:28 AM UTC+2, Robert Deaton wrote:

> Hi all,
> 
> I've been testing out the new completion suggester in master, since it  
> fits a use case I have pretty well. I've been having some consistent  
> problems with slow indexing on my cluster, and have spent a while tweaking  
> to try to remedy them but am now led to the conclusion that it's likely the  
> completion suggester that's causing my slowdowns.
> 
> Here's my setup: I'm indexing approximately 2 billion docs, with two  
> completion fields, each of which is has both input and output length of \<  
> 60 characters.  
> We're using bulk indexing with various batch sizes with little difference  
> in the end results. The index has 0 replicas during indexing, 10 shards,  
> refresh\_interval = -1, warmers disabled. We saw heavy merge volume, so we  
> increased segments\_per\_tier and index buffer size considerably. While we  
> did get some speed improvements, we're still seeing a large number of  
> queries in the indexing\_slowlog.log
> 
> The cluster is 3 master nodes, 10 data nodes, each of which are quad core  
> and 16GB of memory. While running our bulk import scripts, we see the  
> servers remaining at low CPU load and low disk io in iostat. We're using  
> default garbage collector settings (+CondCardMark for Java 7) and we're not  
> seeing large stop the world collections, and our heap usage sits well below  
> the occupancy fraction. We're using the latest Java 7 release on Solaris 10  
> (Joyent's SmartOS).
> 
> In the slowlog, we're seeing a large number docs that are taking anywhere  
> from 500ms to 4s during indexing. A small sample from the slowlog  
> (non-sequential):
> 
> [2013-08-26 22:17:03,626][TRACE][index.indexing.slowlog.index] [qup]  
> [quizlet][8] took[549ms], took\_millis[549], type[words], id[2093528],  
> routing, source[{"id":2093528,"lang\_detect":"tension:en"}]  
> [2013-08-26 22:17:39,694][TRACE][index.indexing.slowlog.index] [qup]  
> [quizlet][8] took[526.3ms], took\_millis[526], type[words], id[2076947],  
> routing,  
> source[{"id":2076947,"lang\_detect":"Puesto:es","lang\_completion":{"input":"es:en:Puesto:poner(to  
> put/ to place)","output":"es:en:Puesto:poner(to put/ to place)"}}]  
> [2013-08-26 22:19:43,308][TRACE][index.indexing.slowlog.index] [qup]  
> [quizlet][8] took[784.7ms], took\_millis[784], type[words], id[2162549],  
> routing, source[{"id":2162549,"lang\_detect":"the  
> teacher:en","lang\_completion":{"input":"en:es:the teacher:la  
> maestra","output":"en:es:the teacher:la maestra"}}]
> 
> Our mapping for this type is as follows:
> 
> ```
> "words" : {
> "_all" : {
> "enabled" : false
> },
> "_source" : {
> "enabled" : false
> },
> "properties" : {
> "id" : {
> "type" : "long"
> },
> "lang_completion" : {
> "type" : "completion",
> "analyzer" : "foldedlowerkeyword",
> "payloads" : false,
> "preserve_separators" : true,
> "preserve_position_increments" : true
> },
> "lang_detect" : {
> "type" : "completion",
> "analyzer" : "foldedlowerkeyword",
> "payloads" : false,
> "preserve_separators" : true,
> "preserve_position_increments" : true
> }
> }
> 
> ```
> 
> Analyzer to match that:
> 
> "index.analysis.analyzer.foldedlowerkeyword.filter.1" : "asciifolding",  
> "index.analysis.analyzer.foldedlowerkeyword.filter.0" : "lowercase",  
> "index.analysis.analyzer.foldedlowerkeyword.tokenizer" : "keyword",  
> "index.analysis.analyzer.foldedlowerkeyword.type" : "custom",
> 
> Any tips on where to look further to diagnose this would be appreciated. At this point we've tweaked enough that I'm wondering about the performance profile of the completion suggestor, since it is a new feature and hasn't seen too much testing. Is there something we can provide to help dig into exactly where we are having slowdowns?
> 
> Thanks,
> 
> --
> 
> Robert Deaton

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:19am UTC](https://discuss.elastic.co/t/completion-suggester-indexing-speed-issues/13336/3 "2017-07-06T02:19:36Z")

</div>