# Aggregation on big data
**URL:** https://discuss.elastic.co/t/aggregation-on-big-data/16709
**Category:** Elasticsearch
**Created:** [March 31, 2014, 9:01am UTC](https://discuss.elastic.co/t/aggregation-on-big-data/16709 "2014-03-31T09:01:28Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@vir\_candy](https://discuss.elastic.co/u/vir_candy)
#### Post date: [March 31, 2014, 9:01am UTC](https://discuss.elastic.co/t/aggregation-on-big-data/16709/1 "2014-03-31T09:01:28Z")
I have 200 million lines of data(about port scanning). I want ES to return
those "ip" who open not only one port at the same time(order by count).
But, considering the volume of data and very little docs have same value on
"ip" field, obviously I get an out of memory error. Is there any way to
finish my query mission.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jpountz](https://discuss.elastic.co/u/jpountz)
#### Post date: [March 31, 2014, 9:26am UTC](https://discuss.elastic.co/t/aggregation-on-big-data/16709/2 "2014-03-31T09:26:09Z")
This kind of use-case requires memory for two main reasons:
- field data,
- counting values (aggregations).
Field data memory usage can be reduced by using doc values[1] which will
effectively store data on disk instead of memory and rely on the filesystem
cache.
Aggregations memory usage is more complicated to improve. In case you are
storing your IPs as string fields, you might want to use the `map`
execution hint that requires less memory than the `ordinals` execution hint
(please however note that we are working on improving the efficiency of
ordinals on high-cardinality fields so it might improve in future versions).
[1]
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
[2]
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
On Mon, Mar 31, 2014 at 11:01 AM, [vir.candy@gmail.com](mailto:vir.candy@gmail.com) wrote:
> I have 200 million lines of data(about port scanning). I want ES to return
> those "ip" who open not only one port at the same time(order by count).
> But, considering the volume of data and very little docs have same value on
> "ip" field, obviously I get an out of memory error. Is there any way to
> finish my query mission.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mS0Z3CSBmBD52v668knp-nR5UpXfYUPC8c4VgAbaMAw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mS0Z3CSBmBD52v668knp-nR5UpXfYUPC8c4VgAbaMAw%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@vir\_candy](https://discuss.elastic.co/u/vir_candy)
#### Post date: [March 31, 2014, 9:37am UTC](https://discuss.elastic.co/t/aggregation-on-big-data/16709/3 "2014-03-31T09:37:10Z")
Thank you!
在 2014年3月31日星期一UTC+8下午5时26分09秒，Adrien Grand写道：
> This kind of use-case requires memory for two main reasons:
>
> - field data,
> - counting values (aggregations).
>
> Field data memory usage can be reduced by using doc values[1] which will
> effectively store data on disk instead of memory and rely on the filesystem
> cache.
>
> Aggregations memory usage is more complicated to improve. In case you are
> storing your IPs as string fields, you might want to use the `map`
> execution hint that requires less memory than the `ordinals` execution hint
> (please however note that we are working on improving the efficiency of
> ordinals on high-cardinality fields so it might improve in future versions).
>
> [1]
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/)
> [2]
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html#_execution_hint)
>
> On Mon, Mar 31, 2014 at 11:01 AM, \<[vir....@gmail.com](mailto:vir....@gmail.com) \\> wrote:
>
> > I have 200 million lines of data(about port scanning). I want ES to
> > return those "ip" who open not only one port at the same time(order by
> > count). But, considering the volume of data and very little docs have same
> > value on "ip" field, obviously I get an out of memory error. Is there any
> > way to finish my query mission.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c15fcb9d-14f0-4eac-ba33-4b46d21c75a0%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> Adrien Grand
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f09ca4ee-d34d-430d-ba58-9ec9136273e2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f09ca4ee-d34d-430d-ba58-9ec9136273e2%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:39am UTC](https://discuss.elastic.co/t/aggregation-on-big-data/16709/4 "2017-07-06T01:39:26Z")