Custom Aggregations

Is there any way we can define our own aggregation functions beyond the
provided metric and bucket aggregations?

Thanks!

Justin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/90accf78-6540-4126-8268-7f997353ad74%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey,

sure, you can write a custom plugin, which implements your custom
aggregations, if you want to.
Any specific functionality you are missing (might be interesting to get it
in the core anyway?).

--Alex

On Tue, Feb 11, 2014 at 9:41 PM, Justin Uang justin.uang@gmail.com wrote:

Is there any way we can define our own aggregation functions beyond the
provided metric and bucket aggregations?

Thanks!

Justin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/90accf78-6540-4126-8268-7f997353ad74%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9_%3DqiV2dBUnMoj4rYkCdbbBCbt6%2Bd5K1w_xh1dBzagSQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

I'm just interested in the flexibility of this platform. How would I
implement the custom aggregation?

On Wednesday, February 12, 2014 8:49:28 AM UTC-5, Alexander Reelsen wrote:

Hey,

sure, you can write a custom plugin, which implements your custom
aggregations, if you want to.
Any specific functionality you are missing (might be interesting to get it
in the core anyway?).

--Alex

On Tue, Feb 11, 2014 at 9:41 PM, Justin Uang <justi...@gmail.com<javascript:>

wrote:

Is there any way we can define our own aggregation functions beyond the
provided metric and bucket aggregations?

Thanks!

Justin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/90accf78-6540-4126-8268-7f997353ad74%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02905d0a-a7a3-465b-9475-e9dedcefd658%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

I'm trying to create a new aggregator. I was able to understand the structure needed to create and register a new aggregator but I don't know who to get the document field values inside collect method. How can I do that?

Hi,

I'm trying to create a new aggregator plugin. I was able to understand the
structure needed to create and register my new aggregator but I don't know
who to get the document field values inside collect method. How can I do
that?

Em quarta-feira, 12 de fevereiro de 2014 11h49min28s UTC-2, Alexander
Reelsen escreveu:

Hey,

sure, you can write a custom plugin, which implements your custom
aggregations, if you want to.
Any specific functionality you are missing (might be interesting to get it
in the core anyway?).

--Alex

On Tue, Feb 11, 2014 at 9:41 PM, Justin Uang <justi...@gmail.com<javascript:>

wrote:

Is there any way we can define our own aggregation functions beyond the
provided metric and bucket aggregations?

Thanks!

Justin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/90accf78-6540-4126-8268-7f997353ad74%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/15e052d8-41d6-48f8-9f29-5482bc4bf8f0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Alex,

Is there an updates on this topic? I want to know how easy or how complex
is it to write a custom aggregator. For example, I want to calculate all
the n-th central moments, with n=2,3,4,..10. By the way, Elasticsearch
already provides the 2nd central moment, i.e. square of the standard
deviation.

In general, I would like to know what kind of custom aggregator is easy to
write?

Thanks,
Jin

On Wednesday, February 12, 2014 5:49:28 AM UTC-8, Alexander Reelsen wrote:

Hey,

sure, you can write a custom plugin, which implements your custom
aggregations, if you want to.
Any specific functionality you are missing (might be interesting to get it
in the core anyway?).

--Alex

On Tue, Feb 11, 2014 at 9:41 PM, Justin Uang <justi...@gmail.com
<javascript:>> wrote:

Is there any way we can define our own aggregation functions beyond the
provided metric and bucket aggregations?

Thanks!

Justin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/90accf78-6540-4126-8268-7f997353ad74%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b2b8894d-c26d-4787-af4c-c45f793befc6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Jin,

It is possible to write a custom aggregation, you can for example look at
this plugin: GitHub - algolia/elasticsearch-cardinality-plugin: This plugin extends Elasticsearch providing a new type of aggregation and a REST action to estimate the cardinality (number of uniq terms) of a field.
that implements a aggregation as a plugin.

If that would work for you, another option would be to contribute this
aggregation to Elaticsearch so that it would be available out of the box
(feel free to open an issue on Github and ping me (@jpountz), I'd be happy
to help).

On Sat, Aug 9, 2014 at 8:19 PM, Jin Huang huang.jin.f@gmail.com wrote:

Hi Alex,

Is there an updates on this topic? I want to know how easy or how complex
is it to write a custom aggregator. For example, I want to calculate all
the n-th central moments, with n=2,3,4,..10. By the way, Elasticsearch
already provides the 2nd central moment, i.e. square of the standard
deviation.

In general, I would like to know what kind of custom aggregator is easy to
write?

Thanks,
Jin

On Wednesday, February 12, 2014 5:49:28 AM UTC-8, Alexander Reelsen wrote:

Hey,

sure, you can write a custom plugin, which implements your custom
aggregations, if you want to.
Any specific functionality you are missing (might be interesting to get
it in the core anyway?).

--Alex

On Tue, Feb 11, 2014 at 9:41 PM, Justin Uang justi...@gmail.com wrote:

Is there any way we can define our own aggregation functions beyond the
provided metric and bucket aggregations?

Thanks!

Justin

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/90accf78-6540-4126-8268-7f997353ad74%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b2b8894d-c26d-4787-af4c-c45f793befc6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b2b8894d-c26d-4787-af4c-c45f793befc6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5pEbWQKAptt10jtwOwuLRXJ5SzLmDY2yn5f_4eqeaY1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

I need a n dimension adjacency matrix.
For example, if I have 3 filters A, B and C , I need to compute
A&B, B&A, C&A but also A&B&C
Unfortunatly, the new adjacency matrix in ES works for 2 dimentions.
I plan to develop a new MultiBucketsAggregation plugin, based on the current adjacency matrix. I now my aggregation will be performant only if I have a few filters, as the number of intersecting buckets will explode for n=3 or more.
Please let me know if my approach is correct.

Regards,

It is, I just want to warn you that maintaining an aggregation is a lot of work as we don't guarantee API backward compatibility of the aggregations API. If your use-case is rather generic, maybe open a Github issue to explain it and we might consider supporting it.

Hi can you please let me know the steps on how to create a custom aggregation.I found some template in src folder in kibana,but where does the logic exist? (for example to perform addition,how does it take two input)

Hi

Can you share any link which can help to write a custom aggregation plugin. I don't even know how to start.Please help

1 Like

Hi, is there any way to use sum aggregation with multiple fields? I don't mean having two (or more) seperate sum aggregations as it takes twice the time as the single aggregation. If that's not possible with the sum aggregation is there any way to write a custom aggregation which resolves this issue (basically sums all the fields and returns an object)?

You may use sum aggregation for each fields, and use bucket script aggregation to sum all.
It will take as long as the number of fields to be aggregated, but it should be faster than creating a custom script.

I was rather thinking of using one sum aggregation and get the same result as I'd get using multiple sum aggregations (I don't need summing up all the fields, I just need something less complex than iterating over each doc with each aggregation - to be precise I'm talking about 50 fields where each sum aggregation takes about 2-3s).

You could try scripted metric aggregation. Though in general, scripts

can’t use Elasticsearch’s index structures or related optimizations. This relationship can sometimes result in slower search speeds.
Scripts, caching, and search speed | Elasticsearch Guide [8.11] | Elastic

it is worth trying and do performance test.

Depending on your use case, you may be able to speed up the process by periodically aggregating the data using transform, or by using the ingest pipeline to pre-calculate the total value of the fields for each document at the time of ingest. Why don't you create a new post with your specific situation and the queries you want (query frequency, filtering conditions...) and discuss it with us?

1 Like

Thx for the answer, tried scripted metric aggregation - it takes twice as long to complete compared to multiple sum aggregations. I might come back some other time with the new post as transforming or pre-calculating values is sth that won't work for us.

1 Like