### Author:  [@dunaeth](https://discuss.elastic.co/u/dunaeth)
#### Post date: [March 10, 2014, 4:24pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/8 "2014-03-10T16:24:27Z")
From now, I'd say the field data size is quite flat whereas the jvm heap
used grows as fielddata\_breaker.estimated\_size\_in\_bytes grows. I'll post
graphs when they'll be relevant.
If it's a jvm heap used issue, could it be due to some kind of caching
issue (though the filter\_cache seems small on each shard) ?
Le lundi 10 mars 2014 09:43:56 UTC+1, Dunaeth a écrit :
> I'm asking our hoster to monitor these metrics and to avoid any confusion,
> the breaker indice size actually monitor the
> fielddata\_breaker.estimated\_size\_in\_bytes from the /\_nodes/stats endpoint.
> Thanks for following this thread 🙂
>
> Le lundi 10 mars 2014 09:34:15 UTC+1, Martijn v Groningen a écrit :
>
> > Yes, the breaker indices size does grow quickly. Can you share the same
> > graphs for jvm heap used and field data size?
> >
> > On 10 March 2014 15:16, Dunaeth [lomig...@gmail.com](mailto:lomig...@gmail.com) wrote:
> >
> > > Hi,
> > >
> > > In order to be more precise, here are the graphs of the metrics we
> > > monitor since we've had the fielddata breaker issue :
> > >
> > > [https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch\_breaker-week.png](https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch_breaker-week.png)
> > >
> > > [https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z\_k/s1600/elasticsearch\_docs-week.png](https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z_k/s1600/elasticsearch_docs-week.png)
> > >
> > > [https://lh3.googleusercontent.com/-M\_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch\_index\_size-week.png](https://lh3.googleusercontent.com/-M_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch_index_size-week.png)
> > >
> > > As one can see, the indices grow kind of linearly with a size which
> > > remains relatively small when the fielddata breaker estimated size grows
> > > exponentially.
> > >
> > > Le jeudi 6 mars 2014 14:49:04 UTC+1, Dunaeth a écrit :
> > >
> > > > At the moment, we have a whole index size of less than 100MB (less than
> > > > 200MB with backuped data) and the estimated\_size is 1.4GB... How are we
> > > > supposed to deal we that kind of trouble ?
> > > >
> > > > Le mardi 4 mars 2014 06:50:56 UTC+1, Dunaeth a écrit :
> > > >
> > > > > Isn't it a bit weird that we reached a 800MB limit and shortcircuited
> > > > > the data processing when our whole indices size is only 140MB (half this
> > > > > size actually since it includes a backup node) ?
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > Met vriendelijke groet,
> >
> > Martijn van Groningen
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8366c5bd-d9e3-49be-9ceb-12190625af1d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8366c5bd-d9e3-49be-9ceb-12190625af1d%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dunaeth](https://discuss.elastic.co/u/dunaeth)
#### Post date: [March 11, 2014, 8:21am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/9 "2014-03-11T08:21:49Z")
Here are the requested metric graphs on our two nodes :
[https://lh5.googleusercontent.com/-kdSq9BgxqGs/Ux7Gz28b\_aI/AAAAAAAAAHM/cggi3HIp5us/s1600/elasticsearch\_breaker-day.png](https://lh5.googleusercontent.com/-kdSq9BgxqGs/Ux7Gz28b_aI/AAAAAAAAAHM/cggi3HIp5us/s1600/elasticsearch_breaker-day.png)
[https://lh6.googleusercontent.com/-k4Of7rvaP0I/Ux7G4BAvKeI/AAAAAAAAAHU/zUVdNuonv0A/s1600/elasticsearch2\_breaker-day.png](https://lh6.googleusercontent.com/-k4Of7rvaP0I/Ux7G4BAvKeI/AAAAAAAAAHU/zUVdNuonv0A/s1600/elasticsearch2_breaker-day.png)
Things that could help to identify the issue is that we didn't had any
trouble with only one index and that it started when we had a statistic
data structure using logstash with month indices. We use these indices to
store data and perform many percolate queries on a specific tester index
(no data, just percolation queries).
Le lundi 10 mars 2014 17:24:27 UTC+1, Dunaeth a écrit :
> From now, I'd say the field data size is quite flat whereas the jvm heap
> used grows as fielddata\_breaker.estimated\_size\_in\_bytes grows. I'll post
> graphs when they'll be relevant.
> If it's a jvm heap used issue, could it be due to some kind of caching
> issue (though the filter\_cache seems small on each shard) ?
>
> Le lundi 10 mars 2014 09:43:56 UTC+1, Dunaeth a écrit :
>
> > I'm asking our hoster to monitor these metrics and to avoid any
> > confusion, the breaker indice size actually monitor the
> > fielddata\_breaker.estimated\_size\_in\_bytes from the /\_nodes/stats endpoint.
> > Thanks for following this thread 🙂
> >
> > Le lundi 10 mars 2014 09:34:15 UTC+1, Martijn v Groningen a écrit :
> >
> > > Yes, the breaker indices size does grow quickly. Can you share the same
> > > graphs for jvm heap used and field data size?
> > >
> > > On 10 March 2014 15:16, Dunaeth [lomig...@gmail.com](mailto:lomig...@gmail.com) wrote:
> > >
> > > > Hi,
> > > >
> > > > In order to be more precise, here are the graphs of the metrics we
> > > > monitor since we've had the fielddata breaker issue :
> > > >
> > > > [https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch\_breaker-week.png](https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch_breaker-week.png)
> > > >
> > > > [https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z\_k/s1600/elasticsearch\_docs-week.png](https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z_k/s1600/elasticsearch_docs-week.png)
> > > >
> > > > [https://lh3.googleusercontent.com/-M\_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch\_index\_size-week.png](https://lh3.googleusercontent.com/-M_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch_index_size-week.png)
> > > >
> > > > As one can see, the indices grow kind of linearly with a size which
> > > > remains relatively small when the fielddata breaker estimated size grows
> > > > exponentially.
> > > >
> > > > Le jeudi 6 mars 2014 14:49:04 UTC+1, Dunaeth a écrit :
> > > >
> > > > > At the moment, we have a whole index size of less than 100MB (less
> > > > > than 200MB with backuped data) and the estimated\_size is 1.4GB... How are
> > > > > we supposed to deal we that kind of trouble ?
> > > > >
> > > > > Le mardi 4 mars 2014 06:50:56 UTC+1, Dunaeth a écrit :
> > > > >
> > > > > > Isn't it a bit weird that we reached a 800MB limit and shortcircuited
> > > > > > the data processing when our whole indices size is only 140MB (half this
> > > > > > size actually since it includes a backup node) ?
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > To view this discussion on the web visit
> > > > > [https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > .
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > Met vriendelijke groet,
> > >
> > > Martijn van Groningen
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c640ff57-1e31-4ad6-823b-ccf2749d2d93%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c640ff57-1e31-4ad6-823b-ccf2749d2d93%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [March 11, 2014, 10:32am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/10 "2014-03-11T10:32:42Z")
The percolator can take up a big part of your jvm heap. How much percolator
queries are loaded (can be seen via cluster stats api)?
It is weird that the fielddata size stays line stays at the bottom, via
which api do you fetch the field data size stat?
On 11 March 2014 15:21, Dunaeth [lomig.poyet@gmail.com](mailto:lomig.poyet@gmail.com) wrote:
> Here are the requested metric graphs on our two nodes :
>
> [https://lh5.googleusercontent.com/-kdSq9BgxqGs/Ux7Gz28b\_aI/AAAAAAAAAHM/cggi3HIp5us/s1600/elasticsearch\_breaker-day.png](https://lh5.googleusercontent.com/-kdSq9BgxqGs/Ux7Gz28b_aI/AAAAAAAAAHM/cggi3HIp5us/s1600/elasticsearch_breaker-day.png)
>
> [https://lh6.googleusercontent.com/-k4Of7rvaP0I/Ux7G4BAvKeI/AAAAAAAAAHU/zUVdNuonv0A/s1600/elasticsearch2\_breaker-day.png](https://lh6.googleusercontent.com/-k4Of7rvaP0I/Ux7G4BAvKeI/AAAAAAAAAHU/zUVdNuonv0A/s1600/elasticsearch2_breaker-day.png)
>
> Things that could help to identify the issue is that we didn't had any
> trouble with only one index and that it started when we had a statistic
> data structure using logstash with month indices. We use these indices to
> store data and perform many percolate queries on a specific tester index
> (no data, just percolation queries).
>
> Le lundi 10 mars 2014 17:24:27 UTC+1, Dunaeth a écrit :
>
> > From now, I'd say the field data size is quite flat whereas the jvm heap
> > used grows as fielddata\_breaker.estimated\_size\_in\_bytes grows. I'll post
> > graphs when they'll be relevant.
> > If it's a jvm heap used issue, could it be due to some kind of caching
> > issue (though the filter\_cache seems small on each shard) ?
> >
> > Le lundi 10 mars 2014 09:43:56 UTC+1, Dunaeth a écrit :
> >
> > > I'm asking our hoster to monitor these metrics and to avoid any
> > > confusion, the breaker indice size actually monitor the
> > > fielddata\_breaker.estimated\_size\_in\_bytes from the /\_nodes/stats
> > > endpoint. Thanks for following this thread 🙂
> > >
> > > Le lundi 10 mars 2014 09:34:15 UTC+1, Martijn v Groningen a écrit :
> > >
> > > > Yes, the breaker indices size does grow quickly. Can you share the same
> > > > graphs for jvm heap used and field data size?
> > > >
> > > > On 10 March 2014 15:16, Dunaeth [lomig...@gmail.com](mailto:lomig...@gmail.com) wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > In order to be more precise, here are the graphs of the metrics we
> > > > > monitor since we've had the fielddata breaker issue :
> > > > >
> > > > > [https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch\_breaker-week.png](https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch_breaker-week.png)
> > > > >
> > > > > [https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z\_k/s1600/elasticsearch\_docs-week.png](https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z_k/s1600/elasticsearch_docs-week.png)
> > > > >
> > > > > [https://lh3.googleusercontent.com/-M\_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch\_index\_size-week.png](https://lh3.googleusercontent.com/-M_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch_index_size-week.png)
> > > > >
> > > > > As one can see, the indices grow kind of linearly with a size which
> > > > > remains relatively small when the fielddata breaker estimated size grows
> > > > > exponentially.
> > > > >
> > > > > Le jeudi 6 mars 2014 14:49:04 UTC+1, Dunaeth a écrit :
> > > > >
> > > > > > At the moment, we have a whole index size of less than 100MB (less
> > > > > > than 200MB with backuped data) and the estimated\_size is 1.4GB... How are
> > > > > > we supposed to deal we that kind of trouble ?
> > > > > >
> > > > > > Le mardi 4 mars 2014 06:50:56 UTC+1, Dunaeth a écrit :
> > > > > >
> > > > > > > Isn't it a bit weird that we reached a 800MB limit and
> > > > > > > shortcircuited the data processing when our whole indices size is only
> > > > > > > 140MB (half this size actually since it includes a backup node) ?
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > > msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%
> > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > > .
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > >
> > > > --
> > > > Met vriendelijke groet,
> > > >
> > > > Martijn van Groningen
--
Met vriendelijke groet,
Martijn van Groningen
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CA%2BA76TxdS48kEpcp\_PfQrxC-Zojrga8mVmApH3XZfknbzf-1Sw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BA76TxdS48kEpcp_PfQrxC-Zojrga8mVmApH3XZfknbzf-1Sw%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dunaeth](https://discuss.elastic.co/u/dunaeth)
#### Post date: [March 11, 2014, 10:39am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/11 "2014-03-11T10:39:47Z")
We have 5 percolator queries registered in our tester index and we
percolate each stat document before insertion in time based indices.
The fielddata size comes from
_nodes.node.indices.fielddata.memory\_size\_in\_bytes_ field from
_/\_nodes/stats_ endpoint.
Le mardi 11 mars 2014 11:32:42 UTC+1, Martijn v Groningen a écrit :
> The percolator can take up a big part of your jvm heap. How much
> percolator queries are loaded (can be seen via cluster stats api)?
> It is weird that the fielddata size stays line stays at the bottom, via
> which api do you fetch the field data size stat?
>
> On 11 March 2014 15:21, Dunaeth \<[lomig...@gmail.com](mailto:lomig...@gmail.com) \\> wrote:
>
> > Here are the requested metric graphs on our two nodes :
> >
> > [https://lh5.googleusercontent.com/-kdSq9BgxqGs/Ux7Gz28b\_aI/AAAAAAAAAHM/cggi3HIp5us/s1600/elasticsearch\_breaker-day.png](https://lh5.googleusercontent.com/-kdSq9BgxqGs/Ux7Gz28b_aI/AAAAAAAAAHM/cggi3HIp5us/s1600/elasticsearch_breaker-day.png)
> >
> > [https://lh6.googleusercontent.com/-k4Of7rvaP0I/Ux7G4BAvKeI/AAAAAAAAAHU/zUVdNuonv0A/s1600/elasticsearch2\_breaker-day.png](https://lh6.googleusercontent.com/-k4Of7rvaP0I/Ux7G4BAvKeI/AAAAAAAAAHU/zUVdNuonv0A/s1600/elasticsearch2_breaker-day.png)
> >
> > Things that could help to identify the issue is that we didn't had any
> > trouble with only one index and that it started when we had a statistic
> > data structure using logstash with month indices. We use these indices to
> > store data and perform many percolate queries on a specific tester index
> > (no data, just percolation queries).
> >
> > Le lundi 10 mars 2014 17:24:27 UTC+1, Dunaeth a écrit :
> >
> > > From now, I'd say the field data size is quite flat whereas the jvm heap
> > > used grows as fielddata\_breaker.estimated\_size\_in\_bytes grows. I'll
> > > post graphs when they'll be relevant.
> > > If it's a jvm heap used issue, could it be due to some kind of caching
> > > issue (though the filter\_cache seems small on each shard) ?
> > >
> > > Le lundi 10 mars 2014 09:43:56 UTC+1, Dunaeth a écrit :
> > >
> > > > I'm asking our hoster to monitor these metrics and to avoid any
> > > > confusion, the breaker indice size actually monitor the
> > > > fielddata\_breaker.estimated\_size\_in\_bytes from the /\_nodes/stats
> > > > endpoint. Thanks for following this thread 🙂
> > > >
> > > > Le lundi 10 mars 2014 09:34:15 UTC+1, Martijn v Groningen a écrit :
> > > >
> > > > > Yes, the breaker indices size does grow quickly. Can you share the
> > > > > same graphs for jvm heap used and field data size?
> > > > >
> > > > > On 10 March 2014 15:16, Dunaeth [lomig...@gmail.com](mailto:lomig...@gmail.com) wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > In order to be more precise, here are the graphs of the metrics we
> > > > > > monitor since we've had the fielddata breaker issue :
> > > > > >
> > > > > > [https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch\_breaker-week.png](https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch_breaker-week.png)
> > > > > >
> > > > > > [https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z\_k/s1600/elasticsearch\_docs-week.png](https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z_k/s1600/elasticsearch_docs-week.png)
> > > > > >
> > > > > > [https://lh3.googleusercontent.com/-M\_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch\_index\_size-week.png](https://lh3.googleusercontent.com/-M_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch_index_size-week.png)
> > > > > >
> > > > > > As one can see, the indices grow kind of linearly with a size which
> > > > > > remains relatively small when the fielddata breaker estimated size grows
> > > > > > exponentially.
> > > > > >
> > > > > > Le jeudi 6 mars 2014 14:49:04 UTC+1, Dunaeth a écrit :
> > > > > >
> > > > > > > At the moment, we have a whole index size of less than 100MB (less
> > > > > > > than 200MB with backuped data) and the estimated\_size is 1.4GB... How are
> > > > > > > we supposed to deal we that kind of trouble ?
> > > > > > >
> > > > > > > Le mardi 4 mars 2014 06:50:56 UTC+1, Dunaeth a écrit :
> > > > > > >
> > > > > > > > Isn't it a bit weird that we reached a 800MB limit and
> > > > > > > > shortcircuited the data processing when our whole indices size is only
> > > > > > > > 140MB (half this size actually since it includes a backup node) ?
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > > > > Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > > > msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%
> > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > > > .
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > >
> > > > > --
> > > > > Met vriendelijke groet,
> > > > >
> > > > > Martijn van Groningen
>
> --
> Met vriendelijke groet,
>
> Martijn van Groningen
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9a305194-e035-48f8-abd0-7709270bd397%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9a305194-e035-48f8-abd0-7709270bd397%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dakrone](https://discuss.elastic.co/u/dakrone)
#### Post date: [March 11, 2014, 11:02am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/12 "2014-03-11T11:02:12Z")
Hi Dunaeth,
Can you attach the output of `curl 'localhost:9200/_nodes/stats?all'`, I
would like to compare the current breaker estimation and the actual field
data usage (they should be the same if no field data loading is currently
happening).
;; Lee
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dd54da43-9b6d-4842-92fb-3a05734314d6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dd54da43-9b6d-4842-92fb-3a05734314d6%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [March 11, 2014, 11:35am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/13 "2014-03-11T11:35:20Z")
5 percolator queries isn't a big deal. I was thinking order magnitude of
100k percolator queries and up.
On 11 March 2014 18:02, Lee Hinman [matthew.hinman@gmail.com](mailto:matthew.hinman@gmail.com) wrote:
> Hi Dunaeth,
>
> Can you attach the output of `curl 'localhost:9200/_nodes/stats?all'`, I
> would like to compare the current breaker estimation and the actual field
> data usage (they should be the same if no field data loading is currently
> happening).
>
> ;; Lee
--
Met vriendelijke groet,
Martijn van Groningen
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CA%2BA76TwAk7dBiCDncJcW0c74dTkY7sPjNNcxh4o\_sX%2B%2B%3DyXV%2BA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BA76TwAk7dBiCDncJcW0c74dTkY7sPjNNcxh4o_sX%2B%2B%3DyXV%2BA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dunaeth](https://discuss.elastic.co/u/dunaeth)
#### Post date: [March 11, 2014, 12:36pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/14 "2014-03-11T12:36:42Z")
Here it is :
{
"cluster\_name": "cluster",
"nodes": {
"G9QD\_\_pjSX6EEAgf0-R6DA": {
"timestamp": 1394541121431,
"name": "Rachel Grey",
"transport\_address": "inet[/10.16.75.4:9300]",
"host": "esnode2",
"ip": [
"inet[/10.16.75.4:9300]",
"NONE"
],
"indices": {
"docs": {
"count": 267760,
"deleted": 27
},
"store": {
"size\_in\_bytes": 117169306,
"throttle\_time\_in\_millis": 529910
},
"indexing": {
"index\_total": 72967,
"index\_time\_in\_millis": 680811,
"index\_current": 0,
"delete\_total": 3361,
"delete\_time\_in\_millis": 350,
"delete\_current": 0
},
"get": {
"total": 0,
"time\_in\_millis": 0,
"exists\_total": 0,
"exists\_time\_in\_millis": 0,
"missing\_total": 0,
"missing\_time\_in\_millis": 0,
"current": 0
},
"search": {
"open\_contexts": 0,
"query\_total": 1867521,
"query\_time\_in\_millis": 24867921,
"query\_current": 0,
"fetch\_total": 513026,
"fetch\_time\_in\_millis": 448526,
"fetch\_current": 0
},
"merges": {
"current": 0,
"current\_docs": 0,
"current\_size\_in\_bytes": 0,
"total": 172,
"total\_time\_in\_millis": 297808,
"total\_docs": 1616101,
"total\_size\_in\_bytes": 480732242
},
"refresh": {
"total": 11450,
"total\_time\_in\_millis": 559302
},
"flush": {
"total": 217,
"total\_time\_in\_millis": 4368
},
"warmer": {
"current": 0,
"total": 12265,
"total\_time\_in\_millis": 28495
},
"filter\_cache": {
"memory\_size\_in\_bytes": 2501878,
"evictions": 0
},
"id\_cache": {
"memory\_size\_in\_bytes": 0
},
"fielddata": {
"memory\_size\_in\_bytes": 2048898,
"evictions": 0
},
"percolate": {
"total": 15659,
"time\_in\_millis": 33899,
"current": 0,
"memory\_size\_in\_bytes": 50602805,
"memory\_size": "48.2mb",
"queries": 5
},
"completion": {
"size\_in\_bytes": 0
},
"segments": {
"count": 46,
"memory\_in\_bytes": 3048144
},
"translog": {
"operations": 289,
"size\_in\_bytes": 0
}
},
"os": {
"timestamp": 1394541121435,
"uptime\_in\_millis": 19092151,
"load\_average": [
1.1,
0.67,
0.61
],
"cpu": {
"sys": 0,
"user": 23,
"idle": 76,
"usage": 23,
"stolen": 0
},
"mem": {
"free\_in\_bytes": 5992665088,
"used\_in\_bytes": 10874990592,
"free\_percent": 71,
"used\_percent": 28,
"actual\_free\_in\_bytes": 12106989568,
"actual\_used\_in\_bytes": 4760666112
},
"swap": {
"used\_in\_bytes": 0,
"free\_in\_bytes": 0
}
},
"process": {
"timestamp": 1394541121435,
"open\_file\_descriptors": 160,
"cpu": {
"percent": 46,
"sys\_in\_millis": 323580,
"user\_in\_millis": 27690580,
"total\_in\_millis": 28014160
},
"mem": {
"resident\_in\_bytes": 1591611392,
"share\_in\_bytes": 34516992,
"total\_virtual\_in\_bytes": 2852478976
}
},
"jvm": {
"timestamp": 1394541121435,
"uptime\_in\_millis": 89061753,
"mem": {
"heap\_used\_in\_bytes": 1237602520,
"heap\_used\_percent": 58,
"heap\_committed\_in\_bytes": 2130051072,
"heap\_max\_in\_bytes": 2130051072,
"non\_heap\_used\_in\_bytes": 54024912,
"non\_heap\_committed\_in\_bytes": 54394880,
"pools": {
"young": {
"used\_in\_bytes": 8136992,
"max\_in\_bytes": 139591680,
"peak\_used\_in\_bytes": 139591680,
"peak\_max\_in\_bytes": 139591680
},
"survivor": {
"used\_in\_bytes": 1696352,
"max\_in\_bytes": 17432576,
"peak\_used\_in\_bytes": 17432576,
"peak\_max\_in\_bytes": 17432576
},
"old": {
"used\_in\_bytes": 1227771384,
"max\_in\_bytes": 1973026816,
"peak\_used\_in\_bytes": 1227771384,
"peak\_max\_in\_bytes": 1973026816
}
}
},
"threads": {
"count": 48,
"peak\_count": 56
},
"gc": {
"collectors": {
"young": {
"collection\_count": 4300,
"collection\_time\_in\_millis": 64401
},
"old": {
"collection\_count": 0,
"collection\_time\_in\_millis": 0
}
}
},
"buffer\_pools": {
"direct": {
"count": 34,
"used\_in\_bytes": 7728976,
"total\_capacity\_in\_bytes": 7728976
},
"mapped": {
"count": 96,
"used\_in\_bytes": 110742186,
"total\_capacity\_in\_bytes": 110742186
}
}
},
"thread\_pool": {
"generic": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 7,
"completed": 11331
},
"index": {
"threads": 2,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 45015
},
"get": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"snapshot": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 5408
},
"merge": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 11882
},
"suggest": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"bulk": {
"threads": 2,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 5760
},
"optimize": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"warmer": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 12265
},
"flush": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 217
},
"search": {
"threads": 6,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 6,
"completed": 2380681
},
"percolate": {
"threads": 2,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 15659
},
"management": {
"threads": 5,
"queue": 0,
"active": 1,
"rejected": 0,
"largest": 5,
"completed": 11655
},
"refresh": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 11465
}
},
"network": {
"tcp": {
"active\_opens": 82783,
"passive\_opens": 28208795,
"curr\_estab": 54,
"in\_segs": 2351809120,
"out\_segs": 3873742660,
"retrans\_segs": 3506870,
"estab\_resets": 103778,
"attempt\_fails": 62,
"in\_errs": 0,
"out\_rsts": 2437332
}
},
"fs": {
"timestamp": 1394541121436,
"total": {
"total\_in\_bytes": 82566774784,
"free\_in\_bytes": 76417433600,
"available\_in\_bytes": 75578605568,
"disk\_reads": 71744,
"disk\_writes": 43233820,
"disk\_io\_op": 43305564,
"disk\_read\_size\_in\_bytes": 4594340864,
"disk\_write\_size\_in\_bytes": 889864847360,
"disk\_io\_size\_in\_bytes": 894459188224,
"disk\_queue": "0",
"disk\_service\_time": "0"
},
"data": [
{
"path": "/home/elasticsearch/nodes/0",
"mount": "/",
"dev": "/dev/vda2",
"total\_in\_bytes": 82566774784,
"free\_in\_bytes": 76417433600,
"available\_in\_bytes": 75578605568,
"disk\_reads": 71744,
"disk\_writes": 43233820,
"disk\_io\_op": 43305564,
"disk\_read\_size\_in\_bytes": 4594340864,
"disk\_write\_size\_in\_bytes": 889864847360,
"disk\_io\_size\_in\_bytes": 894459188224,
"disk\_queue": "0",
"disk\_service\_time": "0"
}
]
},
"transport": {
"server\_open": 26,
"rx\_count": 4037863,
"rx\_size\_in\_bytes": 1371158333,
"tx\_count": 4036008,
"tx\_size\_in\_bytes": 597805806
},
"http": {
"current\_open": 1,
"total\_opened": 104527
},
"fielddata\_breaker": {
"maximum\_size\_in\_bytes": 1704040857,
"maximum\_size": "1.5gb",
"estimated\_size\_in\_bytes": 796577361,
"estimated\_size": "759.6mb",
"overhead": 1.03
}
},
"UaZqZ9h5T5ey0dla2KJVSA": {
"timestamp": 1394541121431,
"name": "Martin Preston",
"transport\_address": "inet[/10.16.75.3:9300]",
"host": "esnode1",
"ip": [
"inet[/10.16.75.3:9300]",
"NONE"
],
"indices": {
"docs": {
"count": 267760,
"deleted": 24
},
"store": {
"size\_in\_bytes": 117222009,
"throttle\_time\_in\_millis": 1721395
},
"indexing": {
"index\_total": 79162,
"index\_time\_in\_millis": 57805,
"index\_current": 0,
"delete\_total": 3952,
"delete\_time\_in\_millis": 421,
"delete\_current": 0
},
"get": {
"total": 0,
"time\_in\_millis": 0,
"exists\_total": 0,
"exists\_time\_in\_millis": 0,
"missing\_total": 0,
"missing\_time\_in\_millis": 0,
"current": 0
},
"search": {
"open\_contexts": 0,
"query\_total": 2058451,
"query\_time\_in\_millis": 576948,
"query\_current": 0,
"fetch\_total": 572130,
"fetch\_time\_in\_millis": 94208,
"fetch\_current": 0
},
"merges": {
"current": 0,
"current\_docs": 0,
"current\_size\_in\_bytes": 0,
"total": 165,
"total\_time\_in\_millis": 60725,
"total\_docs": 1542764,
"total\_size\_in\_bytes": 438737516
},
"refresh": {
"total": 12852,
"total\_time\_in\_millis": 83257
},
"flush": {
"total": 245,
"total\_time\_in\_millis": 3188
},
"warmer": {
"current": 0,
"total": 13792,
"total\_time\_in\_millis": 3157
},
"filter\_cache": {
"memory\_size\_in\_bytes": 2688417,
"evictions": 0
},
"id\_cache": {
"memory\_size\_in\_bytes": 0
},
"fielddata": {
"memory\_size\_in\_bytes": 2231386,
"evictions": 0
},
"percolate": {
"total": 18387,
"time\_in\_millis": 20124,
"current": 0,
"memory\_size\_in\_bytes": 36587613,
"memory\_size": "34.8mb",
"queries": 5
},
"completion": {
"size\_in\_bytes": 0
},
"segments": {
"count": 50,
"memory\_in\_bytes": 3049789
},
"translog": {
"operations": 286,
"size\_in\_bytes": 0
}
},
"os": {
"timestamp": 1394541121432,
"uptime\_in\_millis": 12456460,
"load\_average": [
0.12,
0.11,
0.13
],
"cpu": {
"sys": 2,
"user": 7,
"idle": 90,
"usage": 9,
"stolen": 0
},
"mem": {
"free\_in\_bytes": 7124275200,
"used\_in\_bytes": 9743380480,
"free\_percent": 70,
"used\_percent": 29,
"actual\_free\_in\_bytes": 11817607168,
"actual\_used\_in\_bytes": 5050048512
},
"swap": {
"used\_in\_bytes": 0,
"free\_in\_bytes": 0
}
},
"process": {
"timestamp": 1394541121432,
"open\_file\_descriptors": 160,
"cpu": {
"percent": 3,
"sys\_in\_millis": 325200,
"user\_in\_millis": 2488790,
"total\_in\_millis": 2813990
},
"mem": {
"resident\_in\_bytes": 1869119488,
"share\_in\_bytes": 36052992,
"total\_virtual\_in\_bytes": 2848534528
}
},
"jvm": {
"timestamp": 1394541121432,
"uptime\_in\_millis": 100005398,
"mem": {
"heap\_used\_in\_bytes": 356969744,
"heap\_used\_percent": 16,
"heap\_committed\_in\_bytes": 2130051072,
"heap\_max\_in\_bytes": 2130051072,
"non\_heap\_used\_in\_bytes": 52893472,
"non\_heap\_committed\_in\_bytes": 80510976,
"pools": {
"young": {
"used\_in\_bytes": 17767016,
"max\_in\_bytes": 139591680,
"peak\_used\_in\_bytes": 139591680,
"peak\_max\_in\_bytes": 139591680
},
"survivor": {
"used\_in\_bytes": 2333592,
"max\_in\_bytes": 17432576,
"peak\_used\_in\_bytes": 17432576,
"peak\_max\_in\_bytes": 17432576
},
"old": {
"used\_in\_bytes": 336869136,
"max\_in\_bytes": 1973026816,
"peak\_used\_in\_bytes": 1480030232,
"peak\_max\_in\_bytes": 1973026816
}
}
},
"threads": {
"count": 50,
"peak\_count": 56
},
"gc": {
"collectors": {
"young": {
"collection\_count": 6166,
"collection\_time\_in\_millis": 87399
},
"old": {
"collection\_count": 2,
"collection\_time\_in\_millis": 220
}
}
},
"buffer\_pools": {
"direct": {
"count": 31,
"used\_in\_bytes": 8299659,
"total\_capacity\_in\_bytes": 8299659
},
"mapped": {
"count": 100,
"used\_in\_bytes": 110804161,
"total\_capacity\_in\_bytes": 110804161
}
}
},
"thread\_pool": {
"generic": {
"threads": 2,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 7,
"completed": 16018
},
"index": {
"threads": 2,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 46342
},
"get": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"snapshot": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 6074
},
"merge": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 12779
},
"suggest": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"bulk": {
"threads": 2,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 6719
},
"optimize": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"warmer": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 13792
},
"flush": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 245
},
"search": {
"threads": 6,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 6,
"completed": 2630669
},
"percolate": {
"threads": 2,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 18387
},
"management": {
"threads": 5,
"queue": 0,
"active": 1,
"rejected": 0,
"largest": 5,
"completed": 18982
},
"refresh": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 12852
}
},
"network": {
"tcp": {
"active\_opens": 82615,
"passive\_opens": 53213294,
"curr\_estab": 56,
"in\_segs": 3681090099,
"out\_segs": 3892966411,
"retrans\_segs": 6630780,
"estab\_resets": 68154,
"attempt\_fails": 13,
"in\_errs": 0,
"out\_rsts": 1353371
}
},
"fs": {
"timestamp": 1394541121433,
"total": {
"total\_in\_bytes": 82566774784,
"free\_in\_bytes": 77825974272,
"available\_in\_bytes": 76987146240,
"disk\_reads": 123047,
"disk\_writes": 29374819,
"disk\_io\_op": 29497866,
"disk\_read\_size\_in\_bytes": 7861359616,
"disk\_write\_size\_in\_bytes": 652959784960,
"disk\_io\_size\_in\_bytes": 660821144576,
"disk\_queue": "0",
"disk\_service\_time": "0"
},
"data": [
{
"path": "/home/elasticsearch/nodes/0",
"mount": "/",
"dev": "/dev/vda2",
"total\_in\_bytes": 82566774784,
"free\_in\_bytes": 77825974272,
"available\_in\_bytes": 76987146240,
"disk\_reads": 123047,
"disk\_writes": 29374819,
"disk\_io\_op": 29497866,
"disk\_read\_size\_in\_bytes": 7861359616,
"disk\_write\_size\_in\_bytes": 652959784960,
"disk\_io\_size\_in\_bytes": 660821144576,
"disk\_queue": "0",
"disk\_service\_time": "0"
}
]
},
"transport": {
"server\_open": 26,
"rx\_count": 4449489,
"rx\_size\_in\_bytes": 728136464,
"tx\_count": 4449864,
"tx\_size\_in\_bytes": 1509918122
},
"http": {
"current\_open": 1,
"total\_opened": 7569
},
"fielddata\_breaker": {
"maximum\_size\_in\_bytes": 1704040857,
"maximum\_size": "1.5gb",
"estimated\_size\_in\_bytes": 935746577,
"estimated\_size": "892.3mb",
"overhead": 1.03
}
}
}
}
Le mardi 11 mars 2014 12:02:12 UTC+1, Lee Hinman a écrit :
> Hi Dunaeth,
>
> Can you attach the output of `curl 'localhost:9200/_nodes/stats?all'`, I
> would like to compare the current breaker estimation and the actual field
> data usage (they should be the same if no field data loading is currently
> happening).
>
> ;; Lee
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4aca25cd-2903-4f19-8cf9-09820d73017c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4aca25cd-2903-4f19-8cf9-09820d73017c%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dunaeth](https://discuss.elastic.co/u/dunaeth)
#### Post date: [March 12, 2014, 10:29am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/15 "2014-03-12T10:29:12Z")
Hi,
Latest graphs from this issue, hope it could help :
[https://lh3.googleusercontent.com/-myz7cMtQXlY/UyA22xjbWYI/AAAAAAAAAHk/sKAqPDdbR3Y/s1600/elasticsearch\_breaker-week.png](https://lh3.googleusercontent.com/-myz7cMtQXlY/UyA22xjbWYI/AAAAAAAAAHk/sKAqPDdbR3Y/s1600/elasticsearch_breaker-week.png)
[https://lh4.googleusercontent.com/-iAyUV0CwzcQ/UyA26KVn9HI/AAAAAAAAAHs/t3909UFLO3M/s1600/elasticsearch2\_breaker-week.png](https://lh4.googleusercontent.com/-iAyUV0CwzcQ/UyA26KVn9HI/AAAAAAAAAHs/t3909UFLO3M/s1600/elasticsearch2_breaker-week.png)
Le mardi 11 mars 2014 11:39:47 UTC+1, Dunaeth a écrit :
> We have 5 percolator queries registered in our tester index and we
> percolate each stat document before insertion in time based indices.
> The fielddata size comes from
> _nodes.node.indices.fielddata.memory\_size\_in\_bytes_ field from
> _/\_nodes/stats_ endpoint.
>
> Le mardi 11 mars 2014 11:32:42 UTC+1, Martijn v Groningen a écrit :
>
> > The percolator can take up a big part of your jvm heap. How much
> > percolator queries are loaded (can be seen via cluster stats api)?
> > It is weird that the fielddata size stays line stays at the bottom, via
> > which api do you fetch the field data size stat?
> >
> > On 11 March 2014 15:21, Dunaeth [lomig...@gmail.com](mailto:lomig...@gmail.com) wrote:
> >
> > > Here are the requested metric graphs on our two nodes :
> > >
> > > [https://lh5.googleusercontent.com/-kdSq9BgxqGs/Ux7Gz28b\_aI/AAAAAAAAAHM/cggi3HIp5us/s1600/elasticsearch\_breaker-day.png](https://lh5.googleusercontent.com/-kdSq9BgxqGs/Ux7Gz28b_aI/AAAAAAAAAHM/cggi3HIp5us/s1600/elasticsearch_breaker-day.png)
> > >
> > > [https://lh6.googleusercontent.com/-k4Of7rvaP0I/Ux7G4BAvKeI/AAAAAAAAAHU/zUVdNuonv0A/s1600/elasticsearch2\_breaker-day.png](https://lh6.googleusercontent.com/-k4Of7rvaP0I/Ux7G4BAvKeI/AAAAAAAAAHU/zUVdNuonv0A/s1600/elasticsearch2_breaker-day.png)
> > >
> > > Things that could help to identify the issue is that we didn't had any
> > > trouble with only one index and that it started when we had a statistic
> > > data structure using logstash with month indices. We use these indices to
> > > store data and perform many percolate queries on a specific tester index
> > > (no data, just percolation queries).
> > >
> > > Le lundi 10 mars 2014 17:24:27 UTC+1, Dunaeth a écrit :
> > >
> > > > From now, I'd say the field data size is quite flat whereas the jvm
> > > > heap used grows as fielddata\_breaker.estimated\_size\_in\_bytes grows.
> > > > I'll post graphs when they'll be relevant.
> > > > If it's a jvm heap used issue, could it be due to some kind of caching
> > > > issue (though the filter\_cache seems small on each shard) ?
> > > >
> > > > Le lundi 10 mars 2014 09:43:56 UTC+1, Dunaeth a écrit :
> > > >
> > > > > I'm asking our hoster to monitor these metrics and to avoid any
> > > > > confusion, the breaker indice size actually monitor the
> > > > > fielddata\_breaker.estimated\_size\_in\_bytes from the /\_nodes/stats
> > > > > endpoint. Thanks for following this thread 🙂
> > > > >
> > > > > Le lundi 10 mars 2014 09:34:15 UTC+1, Martijn v Groningen a écrit :
> > > > >
> > > > > > Yes, the breaker indices size does grow quickly. Can you share the
> > > > > > same graphs for jvm heap used and field data size?
> > > > > >
> > > > > > On 10 March 2014 15:16, Dunaeth [lomig...@gmail.com](mailto:lomig...@gmail.com) wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > In order to be more precise, here are the graphs of the metrics we
> > > > > > > monitor since we've had the fielddata breaker issue :
> > > > > > >
> > > > > > > [https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch\_breaker-week.png](https://lh4.googleusercontent.com/-esvzNzxQefM/Ux1ziG-KMoI/AAAAAAAAAGs/xH05cbTpIz0/s1600/elasticsearch_breaker-week.png)
> > > > > > >
> > > > > > > [https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z\_k/s1600/elasticsearch\_docs-week.png](https://lh4.googleusercontent.com/-lJ1ib0PGrj8/Ux1zlcrVUcI/AAAAAAAAAG0/HvQpdTC-z_k/s1600/elasticsearch_docs-week.png)
> > > > > > >
> > > > > > > [https://lh3.googleusercontent.com/-M\_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch\_index\_size-week.png](https://lh3.googleusercontent.com/-M_q2MnRllxE/Ux1zniFzMzI/AAAAAAAAAG8/QQk7PnPc6-c/s1600/elasticsearch_index_size-week.png)
> > > > > > >
> > > > > > > As one can see, the indices grow kind of linearly with a size which
> > > > > > > remains relatively small when the fielddata breaker estimated size grows
> > > > > > > exponentially.
> > > > > > >
> > > > > > > Le jeudi 6 mars 2014 14:49:04 UTC+1, Dunaeth a écrit :
> > > > > > >
> > > > > > > > At the moment, we have a whole index size of less than 100MB (less
> > > > > > > > than 200MB with backuped data) and the estimated\_size is 1.4GB... How are
> > > > > > > > we supposed to deal we that kind of trouble ?
> > > > > > > >
> > > > > > > > Le mardi 4 mars 2014 06:50:56 UTC+1, Dunaeth a écrit :
> > > > > > > >
> > > > > > > > > Isn't it a bit weird that we reached a 800MB limit and
> > > > > > > > > shortcircuited the data processing when our whole indices size is only
> > > > > > > > > 140MB (half this size actually since it includes a backup node) ?
> > > > > > > >
> > > > > > > > --
> > > > > > > > You received this message because you are subscribed to the Google
> > > > > > > > Groups "elasticsearch" group.
> > > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > > > To view this discussion on the web visit
> > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/73526a94-](https://groups.google.com/d/msgid/elasticsearch/73526a94-)
> > > > > > > > b16b-48b6-9a27-526b194b145f%[40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/73526a94-b16b-48b6-9a27-526b194b145f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > > > > .
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > >
> > > > > > --
> > > > > > Met vriendelijke groet,
> > > > > >
> > > > > > Martijn van Groningen
> >
> > --
> > Met vriendelijke groet,
> >
> > Martijn van Groningen
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6f5ea742-9075-4f6a-8a1c-1adfdb532ff4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f5ea742-9075-4f6a-8a1c-1adfdb532ff4%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dunaeth](https://discuss.elastic.co/u/dunaeth)
#### Post date: [March 12, 2014, 4:23pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/16 "2014-03-12T16:23:36Z")
From the latest graphs I have, I'd say the fielddata breaker issue is
directly corelated with data inserts :
[https://lh3.googleusercontent.com/-lEaxxagDO88/UyCJfJQuImI/AAAAAAAAAIY/biXcDmvsFzs/s1600/elasticsearch\_breaker-week.png](https://lh3.googleusercontent.com/-lEaxxagDO88/UyCJfJQuImI/AAAAAAAAAIY/biXcDmvsFzs/s1600/elasticsearch_breaker-week.png)[https://lh6.googleusercontent.com/-ML8sEV0UDWk/UyCJjmybRqI/AAAAAAAAAIg/iSpjv7HCURQ/s1600/elasticsearch2\_breaker-week.png](https://lh6.googleusercontent.com/-ML8sEV0UDWk/UyCJjmybRqI/AAAAAAAAAIg/iSpjv7HCURQ/s1600/elasticsearch2_breaker-week.png)
[https://lh6.googleusercontent.com/-9-hfjWPIaBc/UyCJnA8BsPI/AAAAAAAAAIo/ZabS-W25k48/s1600/elasticsearch\_docs-week.png](https://lh6.googleusercontent.com/-9-hfjWPIaBc/UyCJnA8BsPI/AAAAAAAAAIo/ZabS-W25k48/s1600/elasticsearch_docs-week.png)[https://lh4.googleusercontent.com/-MrNqYSrD4qo/UyCJpoqhYKI/AAAAAAAAAIw/WSrZopKaqLw/s1600/elasticsearch2\_docs-week.png](https://lh4.googleusercontent.com/-MrNqYSrD4qo/UyCJpoqhYKI/AAAAAAAAAIw/WSrZopKaqLw/s1600/elasticsearch2_docs-week.png)
[https://lh3.googleusercontent.com/-Ndaw8zN8mxk/UyCJsybbIuI/AAAAAAAAAI4/lv1WaVhgmPU/s1600/elasticsearch\_index\_size-week.png](https://lh3.googleusercontent.com/-Ndaw8zN8mxk/UyCJsybbIuI/AAAAAAAAAI4/lv1WaVhgmPU/s1600/elasticsearch_index_size-week.png)[https://lh3.googleusercontent.com/-O1FxEiH96yo/UyCJvKzK7lI/AAAAAAAAAJA/NGlmTS4v3HA/s1600/elasticsearch2\_index\_size-week.png](https://lh3.googleusercontent.com/-O1FxEiH96yo/UyCJvKzK7lI/AAAAAAAAAJA/NGlmTS4v3HA/s1600/elasticsearch2_index_size-week.png)
I hope there's a workaround for this fielddata breaker issue.
Le lundi 3 mars 2014 12:51:39 UTC+1, Dunaeth a écrit :
> Hi,
>
> I'd like to know how the fielddata breaker settings are used in a cluster.
> We had a single index in production without any issue but when we added
> some new indices, we started to have issues on the old index and some other
> indices related to fielddata breaker settings :
>
> SearchPhaseExecutionException[Failed to execute phase [query], all shards
> failed; shardFailures {[EQ-GzbelTkqfguEfbElNLA][log-2014-03][0]:
> ElasticsearchException[org.elasticsearch.common.breaker.CircuitBreakingException:
> Data too large, data would be larger than limit of [845047398] bytes];
> nested:
> UncheckedExecutionException[org.elasticsearch.common.breaker.CircuitBreakingException:
> Data too large, data would be larger than limit of [845047398] bytes];
> nested: CircuitBreakingException[Data too large, data would be larger than
> limit of [845047398] bytes]; }{[EQ-GzbelTkqfguEfbElNLA][log-2014-03][1]:
> ElasticsearchException[org.elasticsearch.common.breaker.CircuitBreakingException:
> Data too large, data would be larger than limit of [845047398] bytes];
> nested:
> UncheckedExecutionException[org.elasticsearch.common.breaker.CircuitBreakingException:
> Data too large, data would be larger than limit of [845047398] bytes];
> nested: CircuitBreakingException[Data too large, data would be larger than
> limit of [845047398] bytes]; }]
>
> So, I wonder if it's logical that the old index is affected by the
> addition of some other indices.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/69a6daa9-0940-43be-93fb-ffbb336e3480%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/69a6daa9-0940-43be-93fb-ffbb336e3480%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dunaeth](https://discuss.elastic.co/u/dunaeth)
#### Post date: [March 13, 2014, 7:37am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/17 "2014-03-13T07:37:04Z")
I tried to clear all caches to see if it could help but the fielddata breaker estimated size is still skyrocketing...
If it's not cache issue, and it's linked with our data inserts, I can only think about insert process or percolation queries. Any idea ?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/51f4d2b3-e98e-422d-bf67-76844029c425%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/51f4d2b3-e98e-422d-bf67-76844029c425%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dakrone](https://discuss.elastic.co/u/dakrone)
#### Post date: [March 14, 2014, 2:43am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/18 "2014-03-14T02:43:49Z")
On 3/13/14, 1:37 AM, Dunaeth wrote:
> I tried to clear all caches to see if it could help but the fielddata
> breaker estimated size is still skyrocketing... If it's not cache
> issue, and it's linked with our data inserts, I can only think about
> insert process or percolation queries. Any idea ?
Hi Danaeth,
Can you add the lines:
logger:
indices.fielddata.breaker: TRACE
index.fielddata: TRACE
common.breaker: TRACE
in logging.yml for your elasticsearch configuration and restart the
cluster? This will log information about the breaker estimation and
adjustment. If you can run some queries and attach the logs it would be
helpful in tracking down what's going on.
;; Lee
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53226CE5.3060208%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/53226CE5.3060208%40gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dunaeth](https://discuss.elastic.co/u/dunaeth)
#### Post date: [March 14, 2014, 3:34pm UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/19 "2014-03-14T15:34:32Z")
Our nodes are running with these trace settings atm, is there a prefered
way to provide those logs to you ?
Le vendredi 14 mars 2014 03:43:49 UTC+1, Lee Hinman a écrit :
> On 3/13/14, 1:37 AM, Dunaeth wrote:
>
> > I tried to clear all caches to see if it could help but the fielddata
> > breaker estimated size is still skyrocketing... If it's not cache
> > issue, and it's linked with our data inserts, I can only think about
> > insert process or percolation queries. Any idea ?
>
> Hi Danaeth,
>
> Can you add the lines:
>
> logger:
> indices.fielddata.breaker: TRACE
> index.fielddata: TRACE
> common.breaker: TRACE
>
> in logging.yml for your elasticsearch configuration and restart the
> cluster? This will log information about the breaker estimation and
> adjustment. If you can run some queries and attach the logs it would be
> helpful in tracking down what's going on.
>
> ;; Lee
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fcb17e77-0062-4107-a3c0-d635bc9174dc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fcb17e77-0062-4107-a3c0-d635bc9174dc%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dunaeth](https://discuss.elastic.co/u/dunaeth)
#### Post date: [March 17, 2014, 8:36am UTC](https://discuss.elastic.co/t/fielddata-breaker-question/16104/20 "2014-03-17T08:36:46Z")
Hi,
Due to the insert and search query frequency, it's nearly impossible to get
logs from specific queries. That said, logs attached are extracts of the
logs since the cluster restart and are most probably generated during
document inserts.
Le vendredi 14 mars 2014 16:34:32 UTC+1, Dunaeth a écrit :
> Our nodes are running with these trace settings atm, is there a prefered
> way to provide those logs to you ?
>
> Le vendredi 14 mars 2014 03:43:49 UTC+1, Lee Hinman a écrit :
>
> > On 3/13/14, 1:37 AM, Dunaeth wrote:
> >
> > > I tried to clear all caches to see if it could help but the fielddata
> > > breaker estimated size is still skyrocketing... If it's not cache
> > > issue, and it's linked with our data inserts, I can only think about
> > > insert process or percolation queries. Any idea ?
> >
> > Hi Danaeth,
> >
> > Can you add the lines:
> >
> > logger:
> > indices.fielddata.breaker: TRACE
> > index.fielddata: TRACE
> > common.breaker: TRACE
> >
> > in logging.yml for your elasticsearch configuration and restart the
> > cluster? This will log information about the breaker estimation and
> > adjustment. If you can run some queries and attach the logs it would be
> > helpful in tracking down what's going on.
> >
> > ;; Lee
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/58c1e2ef-830f-4fa4-bbca-6d9cf6255cae%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/58c1e2ef-830f-4fa4-bbca-6d9cf6255cae%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
[Next page](https://discuss.elastic.co/t/fielddata-breaker-question/16104.md?page=2)