# Fielddata circuit breaker problems
**URL:** https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035
**Category:** Elasticsearch
**Created:** [October 2, 2014, 1:29am UTC](https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035 "2014-10-02T01:29:30Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Dave\_Galbraith](https://discuss.elastic.co/u/Dave_Galbraith)
#### Post date: [October 2, 2014, 1:29am UTC](https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035/1 "2014-10-02T01:29:30Z")
Hi! So I have millions and millions of documents in my Elasticsearch, each
one of which has a field called "time". I need the results of my queries to
come back in chronological order. So I put a "sort":{"time":{"order":"asc"}}
in all my queries. This was going great on smaller data sets but then
Elasticsearch started sending me 500s and circuit breaker exceptions
started showing up in the logs with "data for field time would be too
large". So I checked out
[http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html)
and that looks a lot like what I've been seeing: seems like it's trying to
pull all the millions of time values into memory even if they're not
relevant to my query. What are my options for fixing this? I can't
compromise chronological order, it's at the heart of my application. "More
memory" would be a short-term fix but the idea is to scale this thing to
trillions and trillions of points and that's a race I don't want to run.
Can I make these exceptions go away without totally tanking performance?
Thanks!
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Adrian\_Luna](https://discuss.elastic.co/u/Adrian_Luna)
#### Post date: [October 2, 2014, 10:21am UTC](https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035/2 "2014-10-02T10:21:24Z")
Have you tried indexing your data using "doc\_values" as your fielddata
format?
El jueves, 2 de octubre de 2014 03:29:30 UTC+2, Dave Galbraith escribió:
> Hi! So I have millions and millions of documents in my Elasticsearch, each
> one of which has a field called "time". I need the results of my queries to
> come back in chronological order. So I put a
> "sort":{"time":{"order":"asc"}} in all my queries. This was going great
> on smaller data sets but then Elasticsearch started sending me 500s and
> circuit breaker exceptions started showing up in the logs with "data for
> field time would be too large". So I checked out
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html)
> and that looks a lot like what I've been seeing: seems like it's trying to
> pull all the millions of time values into memory even if they're not
> relevant to my query. What are my options for fixing this? I can't
> compromise chronological order, it's at the heart of my application. "More
> memory" would be a short-term fix but the idea is to scale this thing to
> trillions and trillions of points and that's a race I don't want to run.
> Can I make these exceptions go away without totally tanking performance?
> Thanks!
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c2ed3bac-93f7-49ad-aea9-01005aacc1ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2ed3bac-93f7-49ad-aea9-01005aacc1ed%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Dave\_Galbraith](https://discuss.elastic.co/u/Dave_Galbraith)
#### Post date: [October 2, 2014, 8:06pm UTC](https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035/3 "2014-10-02T20:06:11Z")
Wow, that really looks like it'll solve all my problems! I'm not entirely
clear from the docs on where exactly I configure that formatting, though:
can you point me in the right direction? Thanks!
On Thursday, October 2, 2014 3:21:24 AM UTC-7, Adrian Luna wrote:
> Have you tried indexing your data using "doc\_values" as your fielddata
> format?
>
> El jueves, 2 de octubre de 2014 03:29:30 UTC+2, Dave Galbraith escribió:
>
> > Hi! So I have millions and millions of documents in my Elasticsearch,
> > each one of which has a field called "time". I need the results of my
> > queries to come back in chronological order. So I put a
> > "sort":{"time":{"order":"asc"}} in all my queries. This was going great
> > on smaller data sets but then Elasticsearch started sending me 500s and
> > circuit breaker exceptions started showing up in the logs with "data for
> > field time would be too large". So I checked out
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html)
> > and that looks a lot like what I've been seeing: seems like it's trying to
> > pull all the millions of time values into memory even if they're not
> > relevant to my query. What are my options for fixing this? I can't
> > compromise chronological order, it's at the heart of my application. "More
> > memory" would be a short-term fix but the idea is to scale this thing to
> > trillions and trillions of points and that's a race I don't want to run.
> > Can I make these exceptions go away without totally tanking performance?
> > Thanks!
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1511d354-92ae-4b91-a428-8626eaf20a64%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1511d354-92ae-4b91-a428-8626eaf20a64%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Adrian\_Luna](https://discuss.elastic.co/u/Adrian_Luna)
#### Post date: [October 2, 2014, 9:40pm UTC](https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035/4 "2014-10-02T21:40:25Z")
When creating the index, specify a mapping and use field\_data. Take the
example from the documentation and embed it into a normal mapping
definition. However, I realised that it'll only work with
string/numeric/geo\_point, so you may need to use timestamp for your time
field.
Hope it helps.
El jueves, 2 de octubre de 2014 03:29:30 UTC+2, Dave Galbraith escribió:
> Hi! So I have millions and millions of documents in my Elasticsearch, each
> one of which has a field called "time". I need the results of my queries to
> come back in chronological order. So I put a
> "sort":{"time":{"order":"asc"}} in all my queries. This was going great
> on smaller data sets but then Elasticsearch started sending me 500s and
> circuit breaker exceptions started showing up in the logs with "data for
> field time would be too large". So I checked out
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html)
> and that looks a lot like what I've been seeing: seems like it's trying to
> pull all the millions of time values into memory even if they're not
> relevant to my query. What are my options for fixing this? I can't
> compromise chronological order, it's at the heart of my application. "More
> memory" would be a short-term fix but the idea is to scale this thing to
> trillions and trillions of points and that's a race I don't want to run.
> Can I make these exceptions go away without totally tanking performance?
> Thanks!
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8ee0351d-21a7-4fd8-9082-d9eb0b5238d0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8ee0351d-21a7-4fd8-9082-d9eb0b5238d0%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [October 2, 2014, 11:12pm UTC](https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035/5 "2014-10-02T23:12:27Z")
If you really want to sort on a timestamp, use discretization strategy for
better performance.
If you use millisecond resolution timestamp, ES will have to load all the
values of the fields, because they are unique. This is quite massive.
But if you store year, month, day counts, hour counts, minute count etc. in
different fields to the resolution you want (e.g. seconds), you can create
fields with much less unique values.
Then you can sort on multiple fields with very small memory consumption,
something like
"sort": [
{ "year": { "order": "asc" }},
{ "month": { "order": "asc" }},
{ "day": { "order": "asc" }},
{ "hour": { "order": "asc" }},
{ "min": { "order": "asc" }},
{ "sec": { "order": "asc" }}
]
Jörg
On Thu, Oct 2, 2014 at 3:29 AM, Dave Galbraith [david92galbraith@gmail.com](mailto:david92galbraith@gmail.com)
wrote:
> Hi! So I have millions and millions of documents in my Elasticsearch, each
> one of which has a field called "time". I need the results of my queries to
> come back in chronological order. So I put a
> "sort":{"time":{"order":"asc"}} in all my queries. This was going great
> on smaller data sets but then Elasticsearch started sending me 500s and
> circuit breaker exceptions started showing up in the logs with "data for
> field time would be too large". So I checked out
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html)
> and that looks a lot like what I've been seeing: seems like it's trying to
> pull all the millions of time values into memory even if they're not
> relevant to my query. What are my options for fixing this? I can't
> compromise chronological order, it's at the heart of my application. "More
> memory" would be a short-term fix but the idea is to scale this thing to
> trillions and trillions of points and that's a race I don't want to run.
> Can I make these exceptions go away without totally tanking performance?
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGg\_mOioj3EGf9NJy57gCDwNvziEPRviOzaPRu3zDNkDA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGg_mOioj3EGf9NJy57gCDwNvziEPRviOzaPRu3zDNkDA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Dave\_Galbraith](https://discuss.elastic.co/u/Dave_Galbraith)
#### Post date: [October 3, 2014, 12:28am UTC](https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035/6 "2014-10-03T00:28:23Z")
Thanks for the input Jorg but millisecond granularity is
application-critical here. I'm trying to work with the doc\_values fielddata
format for time. Currently I'm sending over ISO strings, and it's parsing
them into dates internally, which I understand will stop me from using
doc\_values. I can turn this off though: are there any dire
performance/workability implications from forcing it to store my dates as
strings and not parse them into dates?
On Thursday, October 2, 2014 4:12:35 PM UTC-7, Jörg Prante wrote:
> If you really want to sort on a timestamp, use discretization strategy for
> better performance.
>
> If you use millisecond resolution timestamp, ES will have to load all the
> values of the fields, because they are unique. This is quite massive.
>
> But if you store year, month, day counts, hour counts, minute count etc.
> in different fields to the resolution you want (e.g. seconds), you can
> create fields with much less unique values.
>
> Then you can sort on multiple fields with very small memory consumption,
> something like
>
> "sort": [
> { "year": { "order": "asc" }},
> { "month": { "order": "asc" }},
> { "day": { "order": "asc" }},
> { "hour": { "order": "asc" }},
> { "min": { "order": "asc" }},
> { "sec": { "order": "asc" }}
> ]
>
> Jörg
>
> On Thu, Oct 2, 2014 at 3:29 AM, Dave Galbraith \<[david92g...@gmail.com](mailto:david92g...@gmail.com)
> \\> wrote:
>
> > Hi! So I have millions and millions of documents in my Elasticsearch,
> > each one of which has a field called "time". I need the results of my
> > queries to come back in chronological order. So I put a
> > "sort":{"time":{"order":"asc"}} in all my queries. This was going great
> > on smaller data sets but then Elasticsearch started sending me 500s and
> > circuit breaker exceptions started showing up in the logs with "data for
> > field time would be too large". So I checked out
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html)
> > and that looks a lot like what I've been seeing: seems like it's trying to
> > pull all the millions of time values into memory even if they're not
> > relevant to my query. What are my options for fixing this? I can't
> > compromise chronological order, it's at the heart of my application. "More
> > memory" would be a short-term fix but the idea is to scale this thing to
> > trillions and trillions of points and that's a race I don't want to run.
> > Can I make these exceptions go away without totally tanking performance?
> > Thanks!
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/65791347-1752-42cd-81e8-e1d05d479d79%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/65791347-1752-42cd-81e8-e1d05d479d79%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Dave\_Galbraith](https://discuss.elastic.co/u/Dave_Galbraith)
#### Post date: [October 3, 2014, 12:44am UTC](https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035/7 "2014-10-03T00:44:20Z")
Actually, the date section of
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
seems to suggest that doc\_values is valid for dates, even though they
aren't mentioned in
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/fielddata-formats.html).
Maybe I can get away with not disabling conversion...
On Thursday, October 2, 2014 5:28:23 PM UTC-7, Dave Galbraith wrote:
> Thanks for the input Jorg but millisecond granularity is
> application-critical here. I'm trying to work with the doc\_values fielddata
> format for time. Currently I'm sending over ISO strings, and it's parsing
> them into dates internally, which I understand will stop me from using
> doc\_values. I can turn this off though: are there any dire
> performance/workability implications from forcing it to store my dates as
> strings and not parse them into dates?
>
> On Thursday, October 2, 2014 4:12:35 PM UTC-7, Jörg Prante wrote:
>
> > If you really want to sort on a timestamp, use discretization strategy
> > for better performance.
> >
> > If you use millisecond resolution timestamp, ES will have to load all the
> > values of the fields, because they are unique. This is quite massive.
> >
> > But if you store year, month, day counts, hour counts, minute count etc.
> > in different fields to the resolution you want (e.g. seconds), you can
> > create fields with much less unique values.
> >
> > Then you can sort on multiple fields with very small memory consumption,
> > something like
> >
> > "sort": [
> > { "year": { "order": "asc" }},
> > { "month": { "order": "asc" }},
> > { "day": { "order": "asc" }},
> > { "hour": { "order": "asc" }},
> > { "min": { "order": "asc" }},
> > { "sec": { "order": "asc" }}
> > ]
> >
> > Jörg
> >
> > On Thu, Oct 2, 2014 at 3:29 AM, Dave Galbraith [david92g...@gmail.com](mailto:david92g...@gmail.com)
> > wrote:
> >
> > > Hi! So I have millions and millions of documents in my Elasticsearch,
> > > each one of which has a field called "time". I need the results of my
> > > queries to come back in chronological order. So I put a
> > > "sort":{"time":{"order":"asc"}} in all my queries. This was going great
> > > on smaller data sets but then Elasticsearch started sending me 500s and
> > > circuit breaker exceptions started showing up in the logs with "data for
> > > field time would be too large". So I checked out
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html)
> > > and that looks a lot like what I've been seeing: seems like it's trying to
> > > pull all the millions of time values into memory even if they're not
> > > relevant to my query. What are my options for fixing this? I can't
> > > compromise chronological order, it's at the heart of my application. "More
> > > memory" would be a short-term fix but the idea is to scale this thing to
> > > trillions and trillions of points and that's a race I don't want to run.
> > > Can I make these exceptions go away without totally tanking performance?
> > > Thanks!
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/60c63662-71b5-4e98-b125-995e357cd06e%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0c28ae08-94ae-4b19-83d5-e8aa0e7fb63a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0c28ae08-94ae-4b19-83d5-e8aa0e7fb63a%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:58am UTC](https://discuss.elastic.co/t/fielddata-circuit-breaker-problems/20035/8 "2017-07-06T00:58:30Z")