Generate multiple scripted fields for the whole index to filter and aggregate

Elastic Stack Elasticsearch
1

I would like to generate multiple scripted fields for the whole index (per each document in the index), for post_filter and aggregate (with filter) by those dynamic values.

As I understand, at the moment the only solution to get the scripted value through all the documents in the index to use score, but it has limitations of:

  • it's not possible to use in the post_filter
  • it's not allowed to have multiple scores

What should I do in such case?

2

You could go a long way with Lucene expressions (because they're fast) and OR filters on the score to store multiple fields on a single score, but you would still bump a wall with the post_filter.

The only possible solution I can think of is some custom metric aggregation, since there you have more control on the map/reduce, but again - I have not experimented with that and it would still be a terrible hack.

3

@gm42 can you please point a link how to use Lucene expressions to store multiple fields on a single score?
I can think of pushing different values within one number as such (100 and 200): 10000000000200

4

I think to replace everything with mongodb, as mongo has aggregation pipeline.

5

Sorry, I was in a meeting and couldn't reply before.

In order to store multiple fields in the score one needs to treat it as a bitfield e.g. you assign bits 0 to 4 for a field, then bits 5 to 9 to another field etc.

This can only work if you need less than the total amount of bits available (which is 53 for an IEEE floating point).

As per aggregation pipeline: ElasticSearch has it too, see https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html