# Fielddata breaker question

**URL:** <https://discuss.elastic.co/t/fielddata-breaker-question/16104>\
**Category:** Elasticsearch\
**Created:** [March 3, 2014, 11:51am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104 "2014-03-03T11:51:39Z")\
**Posts on this page:** 17\
**Page:** 2

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [March 17, 2014, 10:00pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/21 "2014-03-17T22:00:19Z")

</div>

On 3/17/14, 2:36 AM, Dunaeth wrote:

> Hi,
> 
> Due to the insert and search query frequency, it's nearly impossible to  
> get logs from specific queries. That said, logs attached are extracts of  
> the logs since the cluster restart and are most probably generated  
> during document inserts.

Hi Dunaeth,

Thanks! I'll take a look and let you know if I see anything!

;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5327707A.1080608%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5327707A.1080608%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [March 18, 2014, 11:41am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/22 "2014-03-18T11:41:19Z")

</div>

On 3/17/14, 2:36 AM, Dunaeth wrote:

> Hi,
> 
> Due to the insert and search query frequency, it's nearly impossible to  
> get logs from specific queries. That said, logs attached are extracts of  
> the logs since the cluster restart and are most probably generated  
> during document inserts.

It looks like you have incredibly small segments for this index  
(tester), what does the data look like? Can you share your mappings for  
the index as well as example documents?

;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/532830DF.9040905%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/532830DF.9040905%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dunaeth](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@dunaeth](https://discuss.elastic.co/u/dunaeth)\
**Post date:** [March 18, 2014, 11:58am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/23 "2014-03-18T11:58:56Z")

</div>

Actually, tester is a dedicated percolator index with 5 percolation queries  
stored and no other data. Percolated documents are web logs and the tester  
mapping is :

{

> "tester": {  
> "mappings": {  
> ".percolator": {  
> "\_id": {  
> "index": "not\_analyzed"  
> },  
> "properties": {  
> "query": {  
> "type": "object",  
> "enabled": false  
> }  
> }  
> },  
> "test\_hit": {  
> "dynamic\_templates": [  
> {  
> "template1": {  
> "mapping": {  
> "type": "integer"  
> },  
> "match": "\*\_id"  
> }  
> }  
> ],  
> "\_timestamp": {  
> "enabled": true,  
> "path": "date",  
> "format": "date\_time"  
> },  
> "\_source": {  
> "excludes": [  
> "@timestamp"  
> ]  
> },  
> "properties": {  
> "@timestamp": {  
> "type": "date",  
> "index": "no",  
> "format": "dateOptionalTime"  
> },  
> "date": {  
> "type": "date",  
> "format": "date\_time"  
> },  
> "geoip": {  
> "type": "geo\_point"  
> },  
> "host": {  
> "type": "string",  
> "index": "not\_analyzed"  
> },  
> "ip": {  
> "type": "ip"  
> },  
> "prefered-language": {  
> "type": "string"  
> },  
> "referer": {  
> "type": "string",  
> "analyzer": "splitter"  
> },  
> "reverse\_ip": {  
> "type": "string"  
> },  
> "session\_id": {  
> "type": "string",  
> "index": "not\_analyzed"  
> },  
> "ua\_build": {  
> "type": "short"  
> },  
> "ua\_device": {  
> "type": "string"  
> },  
> "ua\_major": {  
> "type": "short"  
> },  
> "ua\_minor": {  
> "type": "short"  
> },  
> "ua\_name": {  
> "type": "string"  
> },  
> "ua\_os": {  
> "type": "string"  
> },  
> "ua\_os\_major": {  
> "type": "short"  
> },  
> "ua\_os\_minor": {  
> "type": "short"  
> },  
> "ua\_os\_name": {  
> "type": "string"  
> },  
> "ua\_patch": {  
> "type": "short"  
> },  
> "unique": {  
> "type": "boolean"  
> },  
> "uri": {  
> "type": "string"  
> },  
> "user-agent": {  
> "type": "string",  
> "analyzer": "splitter"  
> },  
> "valid": {  
> "type": "boolean"  
> }  
> }  
> }  
> }  
> }  
> }

Le mardi 18 mars 2014 12:41:19 UTC+1, Lee Hinman a écrit :

> On 3/17/14, 2:36 AM, Dunaeth wrote:
> 
> > Hi,
> > 
> > Due to the insert and search query frequency, it's nearly impossible to  
> > get logs from specific queries. That said, logs attached are extracts of  
> > the logs since the cluster restart and are most probably generated  
> > during document inserts.
> 
> It looks like you have incredibly small segments for this index  
> (tester), what does the data look like? Can you share your mappings for  
> the index as well as example documents?
> 
> ;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/57ed2777-ee88-47ac-8033-fa87234d1a64%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/57ed2777-ee88-47ac-8033-fa87234d1a64%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dunaeth](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@dunaeth](https://discuss.elastic.co/u/dunaeth)\
**Post date:** [March 18, 2014, 2:07pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/24 "2014-03-18T14:07:24Z")

</div>

That said, our stats indices (monthly indices) have almost the same mapping  
but the documents are stored. I do not believe the tester index is  
concerned by the issue since the only logs linked with it remains in the  
first seconds after the cluster restart (no trace log after). To go further  
with our data description, sure each document remains quiet small (atm  
we're talking about an average of 250B per document in the index size, 2M  
logs for 500MB in size). To be complete, here is the detail of our customer  
splitter analyzer (from tester/\_settings) :

{

> "tester": {  
> "settings": {  
> "index": {  
> "uuid": "HGEPQdWoRLWe0ATNGUElvw",  
> "number\_of\_replicas": "1",  
> "analysis": {  
> "analyzer": {  
> "splitter": {  
> "type": "custom",  
> "tokenizer": "pattern"  
> }  
> }  
> },  
> "number\_of\_shards": "1",  
> "version": {  
> "created": "1000099"  
> }  
> }  
> }  
> }  
> }

Le mardi 18 mars 2014 12:58:56 UTC+1, Dunaeth a écrit :

> Actually, tester is a dedicated percolator index with 5 percolation  
> queries stored and no other data. Percolated documents are web logs and the  
> tester mapping is :
> 
> {
> 
> > "tester": {  
> > "mappings": {  
> > ".percolator": {  
> > "\_id": {  
> > "index": "not\_analyzed"  
> > },  
> > "properties": {  
> > "query": {  
> > "type": "object",  
> > "enabled": false  
> > }  
> > }  
> > },  
> > "test\_hit": {  
> > "dynamic\_templates": [  
> > {  
> > "template1": {  
> > "mapping": {  
> > "type": "integer"  
> > },  
> > "match": "\*\_id"  
> > }  
> > }  
> > ],  
> > "\_timestamp": {  
> > "enabled": true,  
> > "path": "date",  
> > "format": "date\_time"  
> > },  
> > "\_source": {  
> > "excludes": [  
> > "@timestamp"  
> > ]  
> > },  
> > "properties": {  
> > "@timestamp": {  
> > "type": "date",  
> > "index": "no",  
> > "format": "dateOptionalTime"  
> > },  
> > "date": {  
> > "type": "date",  
> > "format": "date\_time"  
> > },  
> > "geoip": {  
> > "type": "geo\_point"  
> > },  
> > "host": {  
> > "type": "string",  
> > "index": "not\_analyzed"  
> > },  
> > "ip": {  
> > "type": "ip"  
> > },  
> > "prefered-language": {  
> > "type": "string"  
> > },  
> > "referer": {  
> > "type": "string",  
> > "analyzer": "splitter"  
> > },  
> > "reverse\_ip": {  
> > "type": "string"  
> > },  
> > "session\_id": {  
> > "type": "string",  
> > "index": "not\_analyzed"  
> > },  
> > "ua\_build": {  
> > "type": "short"  
> > },  
> > "ua\_device": {  
> > "type": "string"  
> > },  
> > "ua\_major": {  
> > "type": "short"  
> > },  
> > "ua\_minor": {  
> > "type": "short"  
> > },  
> > "ua\_name": {  
> > "type": "string"  
> > },  
> > "ua\_os": {  
> > "type": "string"  
> > },  
> > "ua\_os\_major": {  
> > "type": "short"  
> > },  
> > "ua\_os\_minor": {  
> > "type": "short"  
> > },  
> > "ua\_os\_name": {  
> > "type": "string"  
> > },  
> > "ua\_patch": {  
> > "type": "short"  
> > },  
> > "unique": {  
> > "type": "boolean"  
> > },  
> > "uri": {  
> > "type": "string"  
> > },  
> > "user-agent": {  
> > "type": "string",  
> > "analyzer": "splitter"  
> > },  
> > "valid": {  
> > "type": "boolean"  
> > }  
> > }  
> > }  
> > }  
> > }  
> > }
> 
> Le mardi 18 mars 2014 12:41:19 UTC+1, Lee Hinman a écrit :
> 
> > On 3/17/14, 2:36 AM, Dunaeth wrote:
> > 
> > > Hi,
> > > 
> > > Due to the insert and search query frequency, it's nearly impossible to  
> > > get logs from specific queries. That said, logs attached are extracts  
> > > of  
> > > the logs since the cluster restart and are most probably generated  
> > > during document inserts.
> > 
> > It looks like you have incredibly small segments for this index  
> > (tester), what does the data look like? Can you share your mappings for  
> > the index as well as example documents?
> > 
> > ;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/632227b9-d295-45cc-be34-5580d9e793c8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/632227b9-d295-45cc-be34-5580d9e793c8%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dunaeth](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@dunaeth](https://discuss.elastic.co/u/dunaeth)\
**Post date:** [March 18, 2014, 2:10pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/25 "2014-03-18T14:10:55Z")

</div>

Meant "custom" splitter analyzer

Le mardi 18 mars 2014 15:07:24 UTC+1, Dunaeth a écrit :

> That said, our stats indices (monthly indices) have almost the same  
> mapping but the documents are stored. I do not believe the tester index is  
> concerned by the issue since the only logs linked with it remains in the  
> first seconds after the cluster restart (no trace log after). To go further  
> with our data description, sure each document remains quiet small (atm  
> we're talking about an average of 250B per document in the index size, 2M  
> logs for 500MB in size). To be complete, here is the detail of our customer  
> splitter analyzer (from tester/\_settings) :
> 
> {
> 
> > "tester": {  
> > "settings": {  
> > "index": {  
> > "uuid": "HGEPQdWoRLWe0ATNGUElvw",  
> > "number\_of\_replicas": "1",  
> > "analysis": {  
> > "analyzer": {  
> > "splitter": {  
> > "type": "custom",  
> > "tokenizer": "pattern"  
> > }  
> > }  
> > },  
> > "number\_of\_shards": "1",  
> > "version": {  
> > "created": "1000099"  
> > }  
> > }  
> > }  
> > }  
> > }
> 
> Le mardi 18 mars 2014 12:58:56 UTC+1, Dunaeth a écrit :
> 
> > Actually, tester is a dedicated percolator index with 5 percolation  
> > queries stored and no other data. Percolated documents are web logs and the  
> > tester mapping is :
> > 
> > {
> > 
> > > "tester": {  
> > > "mappings": {  
> > > ".percolator": {  
> > > "\_id": {  
> > > "index": "not\_analyzed"  
> > > },  
> > > "properties": {  
> > > "query": {  
> > > "type": "object",  
> > > "enabled": false  
> > > }  
> > > }  
> > > },  
> > > "test\_hit": {  
> > > "dynamic\_templates": [  
> > > {  
> > > "template1": {  
> > > "mapping": {  
> > > "type": "integer"  
> > > },  
> > > "match": "\*\_id"  
> > > }  
> > > }  
> > > ],  
> > > "\_timestamp": {  
> > > "enabled": true,  
> > > "path": "date",  
> > > "format": "date\_time"  
> > > },  
> > > "\_source": {  
> > > "excludes": [  
> > > "@timestamp"  
> > > ]  
> > > },  
> > > "properties": {  
> > > "@timestamp": {  
> > > "type": "date",  
> > > "index": "no",  
> > > "format": "dateOptionalTime"  
> > > },  
> > > "date": {  
> > > "type": "date",  
> > > "format": "date\_time"  
> > > },  
> > > "geoip": {  
> > > "type": "geo\_point"  
> > > },  
> > > "host": {  
> > > "type": "string",  
> > > "index": "not\_analyzed"  
> > > },  
> > > "ip": {  
> > > "type": "ip"  
> > > },  
> > > "prefered-language": {  
> > > "type": "string"  
> > > },  
> > > "referer": {  
> > > "type": "string",  
> > > "analyzer": "splitter"  
> > > },  
> > > "reverse\_ip": {  
> > > "type": "string"  
> > > },  
> > > "session\_id": {  
> > > "type": "string",  
> > > "index": "not\_analyzed"  
> > > },  
> > > "ua\_build": {  
> > > "type": "short"  
> > > },  
> > > "ua\_device": {  
> > > "type": "string"  
> > > },  
> > > "ua\_major": {  
> > > "type": "short"  
> > > },  
> > > "ua\_minor": {  
> > > "type": "short"  
> > > },  
> > > "ua\_name": {  
> > > "type": "string"  
> > > },  
> > > "ua\_os": {  
> > > "type": "string"  
> > > },  
> > > "ua\_os\_major": {  
> > > "type": "short"  
> > > },  
> > > "ua\_os\_minor": {  
> > > "type": "short"  
> > > },  
> > > "ua\_os\_name": {  
> > > "type": "string"  
> > > },  
> > > "ua\_patch": {  
> > > "type": "short"  
> > > },  
> > > "unique": {  
> > > "type": "boolean"  
> > > },  
> > > "uri": {  
> > > "type": "string"  
> > > },  
> > > "user-agent": {  
> > > "type": "string",  
> > > "analyzer": "splitter"  
> > > },  
> > > "valid": {  
> > > "type": "boolean"  
> > > }  
> > > }  
> > > }  
> > > }  
> > > }  
> > > }
> > 
> > Le mardi 18 mars 2014 12:41:19 UTC+1, Lee Hinman a écrit :
> > 
> > > On 3/17/14, 2:36 AM, Dunaeth wrote:
> > > 
> > > > Hi,
> > > > 
> > > > Due to the insert and search query frequency, it's nearly impossible  
> > > > to  
> > > > get logs from specific queries. That said, logs attached are extracts  
> > > > of  
> > > > the logs since the cluster restart and are most probably generated  
> > > > during document inserts.
> > > 
> > > It looks like you have incredibly small segments for this index  
> > > (tester), what does the data look like? Can you share your mappings for  
> > > the index as well as example documents?
> > > 
> > > ;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ced2a531-d2f8-4abe-af3d-be896c826a39%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ced2a531-d2f8-4abe-af3d-be896c826a39%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [March 19, 2014, 10:01am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/26 "2014-03-19T10:01:16Z")

</div>

On 3/18/14, 5:58 AM, Dunaeth wrote:

> Actually, tester is a dedicated percolator index with 5 percolation  
> queries stored and no other data. Percolated documents are web logs and  
> the tester mapping is (elided)

Are you doing aggregations when you percolate documents? Are you  
percolating existing documents or sending new ones every time?

;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53296AEC.9010405%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/53296AEC.9010405%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dunaeth](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@dunaeth](https://discuss.elastic.co/u/dunaeth)\
**Post date:** [March 19, 2014, 10:25am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/27 "2014-03-19T10:25:51Z")

</div>

There's no aggregation in percolation queries and we only percolate new  
documents, the flow is as follow :

Step 1 : data logging

1. An event occures
2. possible ES search queries with or without facet on different indices
3. ES percolation query with event data on the tester index (5 queries  
stored, no other data)
4. event data logging in a plain text file

Step 2 : defered data indexing

1. logstash detects the new event and send it to a centralized redis queue  
like the centralized example described in logstash documentation
2. logstash parses the queue and indexes the event in ES

Meanwhile, there can be aggregation queries on the stat part of the  
application but their rate is minimal compared to the insert rate.

Le mercredi 19 mars 2014 11:01:16 UTC+1, Lee Hinman a écrit :

> On 3/18/14, 5:58 AM, Dunaeth wrote:
> 
> > Actually, tester is a dedicated percolator index with 5 percolation  
> > queries stored and no other data. Percolated documents are web logs and  
> > the tester mapping is (elided)
> 
> Are you doing aggregations when you percolate documents? Are you  
> percolating existing documents or sending new ones every time?
> 
> ;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/596f7efa-11e6-4045-a4bc-9200cb535b37%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/596f7efa-11e6-4045-a4bc-9200cb535b37%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [March 20, 2014, 2:41am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/28 "2014-03-20T02:41:27Z")

</div>

On 3/19/14, 4:25 AM, Dunaeth wrote:

> There's no aggregation in percolation queries and we only percolate new  
> documents, the flow is as follow :
> 
> Step 1 : data logging
> 
> 1. An event occures
> 2. possible ES search queries with or without facet on different indices
> 3. ES percolation query with event data on the tester index (5 queries  
> stored, no other data)
> 4. event data logging in a plain text file
> 
> Step 2 : defered data indexing
> 
> 1. logstash detects the new event and send it to a centralized redis  
> queue like the centralized example described in logstash documentation
> 2. logstash parses the queue and indexes the event in ES
> 
> Meanwhile, there can be aggregation queries on the stat part of the  
> application but their rate is minimal compared to the insert rate.

Okay, so we'll rule out percolation for now as a cause of the circuit  
breaker estimations.

More information about your data would be helpful, you sent the mapping,  
can you send an example document that is similar to most of the  
documents you're indexing? Can you also provide some of the queries,  
facets, and aggregations that you're performing? I can try to index some  
test data and reproduce it with this information and see if that works.

Also, if this is not sensitive data, taking a snapshot of the index with  
the snapshot/restore functionality and sending it to me would allow me  
to reproduce the issue with your exact data. If that's an option it  
would definitely be useful.

;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/532A5557.7010301%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/532A5557.7010301%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dunaeth](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@dunaeth](https://discuss.elastic.co/u/dunaeth)\
**Post date:** [March 24, 2014, 1:46pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/29 "2014-03-24T13:46:29Z")

</div>

Hi,

Here's a small sample of data (atm, the march index is near 6M docs for  
1.37GB size).

Le jeudi 20 mars 2014 03:41:27 UTC+1, Lee Hinman a écrit :

> On 3/19/14, 4:25 AM, Dunaeth wrote:
> 
> > There's no aggregation in percolation queries and we only percolate new  
> > documents, the flow is as follow :
> > 
> > Step 1 : data logging
> > 
> > 1. An event occures
> > 2. possible ES search queries with or without facet on different indices
> > 3. ES percolation query with event data on the tester index (5 queries  
> > stored, no other data)
> > 4. event data logging in a plain text file
> > 
> > Step 2 : defered data indexing
> > 
> > 1. logstash detects the new event and send it to a centralized redis  
> > queue like the centralized example described in logstash documentation
> > 2. logstash parses the queue and indexes the event in ES
> > 
> > Meanwhile, there can be aggregation queries on the stat part of the  
> > application but their rate is minimal compared to the insert rate.
> 
> Okay, so we'll rule out percolation for now as a cause of the circuit  
> breaker estimations.
> 
> More information about your data would be helpful, you sent the mapping,  
> can you send an example document that is similar to most of the  
> documents you're indexing? Can you also provide some of the queries,  
> facets, and aggregations that you're performing? I can try to index some  
> test data and reproduce it with this information and see if that works.
> 
> Also, if this is not sensitive data, taking a snapshot of the index with  
> the snapshot/restore functionality and sending it to me would allow me  
> to reproduce the issue with your exact data. If that's an option it  
> would definitely be useful.
> 
> ;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/40486946-9167-4d25-9845-60525a96f876%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/40486946-9167-4d25-9845-60525a96f876%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [March 26, 2014, 9:39pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/30 "2014-03-26T21:39:35Z")

</div>

On 3/24/14, 7:46 AM, Dunaeth wrote:

> Hi,
> 
> Here's a small sample of data (atm, the march index is near 6M docs for  
> 1.37GB size).

I haven't been able to reproduce the issue locally, but Simon noticed  
that this may be caused by  
[[LUCENE-5553] IndexReader#ReaderClosedListener is not always called on IndexReader#close() - ASF JIRA](https://issues.apache.org/jira/browse/LUCENE-5553) , which is fixed in  
the upcoming 4.7.1 release (which will be incorporated into ES shortly  
after).

;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53334917.5070302%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/53334917.5070302%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dunaeth](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@dunaeth](https://discuss.elastic.co/u/dunaeth)\
**Post date:** [March 27, 2014, 3:53pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/31 "2014-03-27T15:53:18Z")

</div>

According to the graph, the fielddata\_breaker\_estimated\_size increase is  
linked to our data inserts.  
Since you weren't able to reproduce the issue by inserting given data,  
either the data set is not enough varied or the issue is due to queries  
done prior the inserts.  
For the 1st case, I could probably give you some snapeshots from our data.  
For the 2nd, I could give you more details about our queries.

Le mercredi 26 mars 2014 22:39:35 UTC+1, Lee Hinman a écrit :

> On 3/24/14, 7:46 AM, Dunaeth wrote:
> 
> > Hi,
> > 
> > Here's a small sample of data (atm, the march index is near 6M docs for  
> > 1.37GB size).
> 
> I haven't been able to reproduce the issue locally, but Simon noticed  
> that this may be caused by  
> [[LUCENE-5553] IndexReader#ReaderClosedListener is not always called on IndexReader#close() - ASF JIRA](https://issues.apache.org/jira/browse/LUCENE-5553) , which is fixed in  
> the upcoming 4.7.1 release (which will be incorporated into ES shortly  
> after).
> 
> ;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c1d05b77-d66c-427c-a59a-519208547a2a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c1d05b77-d66c-427c-a59a-519208547a2a%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dunaeth](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@dunaeth](https://discuss.elastic.co/u/dunaeth)\
**Post date:** [April 4, 2014, 2:51pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/32 "2014-04-04T14:51:38Z")

</div>

Hi,

I stopped logstash to check whether the fielddata breaker issue was due to  
data inserts or not. As the fielddata breaker estimated size continued to  
grow the same way, I'd say the issue is caused by the queries done prior  
inserts (ie. eventually a facet or an aggregation query followed by a  
search query (or nothing at all) and finally a percolation query (this one  
happens each time) per event created).

Le jeudi 27 mars 2014 16:53:18 UTC+1, Dunaeth a écrit :

> According to the graph, the fielddata\_breaker\_estimated\_size increase is  
> linked to our data inserts.  
> Since you weren't able to reproduce the issue by inserting given data,  
> either the data set is not enough varied or the issue is due to queries  
> done prior the inserts.  
> For the 1st case, I could probably give you some snapeshots from our data.  
> For the 2nd, I could give you more details about our queries.
> 
> Le mercredi 26 mars 2014 22:39:35 UTC+1, Lee Hinman a écrit :
> 
> > On 3/24/14, 7:46 AM, Dunaeth wrote:
> > 
> > > Hi,
> > > 
> > > Here's a small sample of data (atm, the march index is near 6M docs for  
> > > 1.37GB size).
> > 
> > I haven't been able to reproduce the issue locally, but Simon noticed  
> > that this may be caused by  
> > [[LUCENE-5553] IndexReader#ReaderClosedListener is not always called on IndexReader#close() - ASF JIRA](https://issues.apache.org/jira/browse/LUCENE-5553) , which is fixed in  
> > the upcoming 4.7.1 release (which will be incorporated into ES shortly  
> > after).
> > 
> > ;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5a38a03a-c4b8-4a34-a455-503132dc64e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5a38a03a-c4b8-4a34-a455-503132dc64e4%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [April 4, 2014, 10:54pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/33 "2014-04-04T22:54:10Z")

</div>

On 4/4/14, 8:51 AM, Dunaeth wrote:

> Hi,
> 
> I stopped logstash to check whether the fielddata breaker issue was due  
> to data inserts or not. As the fielddata breaker estimated size  
> continued to grow the same way, I'd say the issue is caused by the  
> queries done prior inserts (ie. eventually a facet or an aggregation  
> query followed by a search query (or nothing at all) and finally a  
> percolation query (this one happens each time) per event created).

Hi Dunaeth,

I think your issue may also be caused by  
[Percolator doesn't reduce CircuitBreaker stats in every case. by martijnvg · Pull Request #5588 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/5588), which was  
recently resolved.

;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/533F3812.5060502%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/533F3812.5060502%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dunaeth](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@dunaeth](https://discuss.elastic.co/u/dunaeth)\
**Post date:** [April 7, 2014, 7:35am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/34 "2014-04-07T07:35:00Z")

</div>

Hi Lee,

This issue could exactly match what we're experiencing, we'll wait for the  
next revision then and see if it solves our problem. Thanks 🙂

Le samedi 5 avril 2014 00:54:10 UTC+2, Lee Hinman a écrit :

> On 4/4/14, 8:51 AM, Dunaeth wrote:
> 
> > Hi,
> > 
> > I stopped logstash to check whether the fielddata breaker issue was due  
> > to data inserts or not. As the fielddata breaker estimated size  
> > continued to grow the same way, I'd say the issue is caused by the  
> > queries done prior inserts (ie. eventually a facet or an aggregation  
> > query followed by a search query (or nothing at all) and finally a  
> > percolation query (this one happens each time) per event created).
> 
> Hi Dunaeth,
> 
> I think your issue may also be caused by  
> [Percolator doesn't reduce CircuitBreaker stats in every case. by martijnvg · Pull Request #5588 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/5588), which was  
> recently resolved.
> 
> ;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d3449b92-590c-4a5f-8e98-47ee4a569518%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d3449b92-590c-4a5f-8e98-47ee4a569518%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dunaeth](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@dunaeth](https://discuss.elastic.co/u/dunaeth)\
**Post date:** [April 17, 2014, 10:32am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/35 "2014-04-17T10:32:18Z")

</div>

Hi,

Some feedback on this subject, the latest ES patches made my day. Using ES  
1.0.3 solved the issue. Thanks 🙂

Le lundi 7 avril 2014 09:35:00 UTC+2, Dunaeth a écrit :

> Hi Lee,
> 
> This issue could exactly match what we're experiencing, we'll wait for the  
> next revision then and see if it solves our problem. Thanks 🙂
> 
> Le samedi 5 avril 2014 00:54:10 UTC+2, Lee Hinman a écrit :
> 
> > On 4/4/14, 8:51 AM, Dunaeth wrote:
> > 
> > > Hi,
> > > 
> > > I stopped logstash to check whether the fielddata breaker issue was due  
> > > to data inserts or not. As the fielddata breaker estimated size  
> > > continued to grow the same way, I'd say the issue is caused by the  
> > > queries done prior inserts (ie. eventually a facet or an aggregation  
> > > query followed by a search query (or nothing at all) and finally a  
> > > percolation query (this one happens each time) per event created).
> > 
> > Hi Dunaeth,
> > 
> > I think your issue may also be caused by  
> > [Percolator doesn't reduce CircuitBreaker stats in every case. by martijnvg · Pull Request #5588 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/5588), which was  
> > recently resolved.
> > 
> > ;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1ccdc25b-90a3-4399-b94a-937299b9ae6c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1ccdc25b-90a3-4399-b94a-937299b9ae6c%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\
**Post date:** [April 17, 2014, 2:39pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/36 "2014-04-17T14:39:06Z")

</div>

On 4/17/14, 4:32 AM, Dunaeth wrote:

> Hi,
> 
> Some feedback on this subject, the latest ES patches made my day. Using  
> ES 1.0.3 solved the issue. Thanks 🙂

Great! Glad to hear it worked for you!

;; Lee

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/534FE78A.3020803%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/534FE78A.3020803%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:35am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/37 "2017-07-06T01:35:14Z")

</div>

Closed on July 6, 2017, 1:35am UTC



[Previous page](https://discuss.elastic.co/t/fielddata-breaker-question/16104.md?page=1)