# Significant terms aggregation custom score with more than one background **URL:** \ **Category:** Elasticsearch\ **Created:** [November 21, 2019, 2:41pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910 "2019-11-21T14:41:51Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![dbuijs](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@dbuijs](https://discuss.elastic.co/u/dbuijs)\ **Post date:** [November 21, 2019, 2:41pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/1 "2019-11-21T14:41:51Z")
I'm trying to implement different types of disproportionality metrics for pharmacovigilance reports in Elasticsearch. These are basically different flavours of significant terms aggregations, and I'm pretty sure I can use custom scoring functions, but I need marginal counts. The individual documents in elastic are adverse drug reports that have some report meta-data, at least one suspect drug, and at least one reaction term. I would aggregate first by suspect drug, and then do significant terms aggregation on reactions within each suspect drug bucket. The problem is that I need access to two different background counts for the scoring function. I need the count of all documents without the suspect drug and the count of all documents without the specific reaction that is being scored. I know I can specify a custom background, but I can't seem to define more than one. Does anyone have any ideas on how something like this could be implemented? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 21, 2019, 3:07pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/2 "2019-11-21T15:07:58Z")
Hi dbuijs, First I'd like to check my assumptions - you want to identify significant reactions for each drug? So the drug is the bucketing that defines subsets of interest and in each of these you are trying to identify only the significant reactions? The significance scores for each reaction/drug combo will be given four numbers: 1. Foreground size - the number of reports for a particular drug 2. Foreground count - the number of reports for a particular reaction for a particular drug 3. Background size - the number of all reports 4. Background count - the number of all reports with a particular reaction > I need the count of all documents without the suspect drug and the count of all documents without the specific reaction that is being scored. Can these be derived by a scripted heuristic from the 4 numbers presented to the scoring function? The query I've used previously for reaction data like this is: ``` GET fdadrugs/_search { "size":0, "aggs": { "drug": { "terms": { "field":"drugs.keyword" }, "aggs":{ "significant_reactions":{ "significant_terms": { "field": "reactions.keyword" } } } } } } ``` ---
**Author:** ![dbuijs](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@dbuijs](https://discuss.elastic.co/u/dbuijs)\ **Post date:** [November 21, 2019, 3:12pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/3 "2019-11-21T15:12:30Z")
Mark, Thanks so much for this. I think that's exactly what I was looking for. It was just way simpler than I thought. Background count for the particular reaction is across all documents in the index, correct? Not just within each drug bucket? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 21, 2019, 3:16pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/4 "2019-11-21T15:16:47Z")
> [@dbuijs](#): > > Background count for the particular reaction is across all documents in the index, correct? Yes. So using that "headache" might be something we can spot as "commonly common" across all drugs and ignore if it was prevalent in any one drug's set of reports. It is the things that are very rare in the background (eg "blood in stools") and comparatively common in the foreground set for a drug that are of most interest. This [visualisation](https://www.youtube.com/watch?v=azP15yvbOBA) may be illuminating. The items in the top left corner are the most interesting. ---
**Author:** ![dbuijs](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@dbuijs](https://discuss.elastic.co/u/dbuijs)\ **Post date:** [November 21, 2019, 3:59pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/5 "2019-11-21T15:59:05Z")
Can I just clarify the terminology here: Foreground size = subset size Foreground count = subset freq Background size = superset size Background count = superset freq Did I get that right? I think I've got something sorted out, but it's barfing due to divide by zero. Is there a way to just ignore or assign these values a null? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 21, 2019, 4:04pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/6 "2019-11-21T16:04:56Z")
> [@dbuijs](#): > > Can I just clarify the terminology here: Yep. > [@dbuijs](#): > > I think I've got something sorted out, but it's barfing due to divide by zero. If you're using a custom background\_filter then your subset may no longer be a subset of the superset. This could produce superset\_freq values of zero if your custom filter matches entirely different sets of docs to the subset. Some of the scoring heuristics have settings to include safeguards where your background filter make results that break the subset/superset assumptions. ---
**Author:** ![dbuijs](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@dbuijs](https://discuss.elastic.co/u/dbuijs)\ **Post date:** [November 21, 2019, 4:55pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/7 "2019-11-21T16:55:46Z")
Here's what I'm using to get the Proportional Reporting Ratio (PRR) ([http://openvigil.sourceforge.net/doc/DPA.pdf](http://openvigil.sourceforge.net/doc/DPA.pdf)): ``` POST drug_event/_search?size=0 { "aggregations": { "drug": { "terms": { "field":"report_ingredient_suspect.keyword" }, "aggs":{ "significant_reactions":{ "significant_terms": { "field": "reaction_pt.keyword", "script_heuristic": { "script": "(params._subset_freq/params._subset_size)/((params._superset_size - params._subset_size)/(params._superset_freq - params._subset_freq))" } } } } } } } ``` And it's giving me a divide by zero error. Any thoughts? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 21, 2019, 5:02pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/8 "2019-11-21T17:02:02Z")
> [@dbuijs](#): > > /(params.\_superset\_freq - params.\_subset\_freq) If a single drug's reports contain all of the incidents of a particular reaction then `superset_freq == subset_freq` and subtracting them gives zero? ---
**Author:** ![dbuijs](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@dbuijs](https://discuss.elastic.co/u/dbuijs)\ **Post date:** [November 21, 2019, 8:49pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/9 "2019-11-21T20:49:06Z")
That does seem to have something to do with it. When I add +1 to the denominator, it doesn't error any more, but I don't get any significant terms. I ran it again with the script just for each individual parameter and that works, but as soon as I try to do something with them, I get empty buckets. Is there a minimum score threshold I can adjust? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 21, 2019, 9:38pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/10 "2019-11-21T21:38:52Z")
I’m not sure I understand your current problem. If your script returns \>0 and doesn’t divide by zero then you should see some results. Have you determined the out of the box heuristics don’t work for you? They all work with the same 4 numbers and generally just have a different emphasis on scoring for popular vs rare terms. Some of the differences are very subtle. ---
**Author:** ![dbuijs](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@dbuijs](https://discuss.elastic.co/u/dbuijs)\ **Post date:** [November 22, 2019, 12:21am UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/11 "2019-11-22T00:21:19Z")
I think I got it working. These particular scoring algorithms have been published and validated in the scientific literature specifically for pharmacovigilance. I do want to look at how the other built-in metrics compare, but in order to get my scientist colleagues to look at it, I need to be able to demonstrate that Elasticsearch is accurately replicating metrics they already understand. We're still seeing some strangeness in the \_superset\_size parameter. It seems to change based on the significant terms aggregation size parameter. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 22, 2019, 8:06am UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/12 "2019-11-22T08:06:49Z")
> [@dbuijs](#): > > It seems to change based on the significant terms aggregation size parameter. This is likely caused by having multiple shards - distribution is often the enemy of accuracy. When your data is not all in one place for analysis then different shards can have different ideas about what the top N terms are. When their results are glued together it’s not necessarily the same answer as the result for the same query on a single shard. The shard\_size setting can help but for accurate significant terms analysis having a single index/shard is best. ---
**Author:** ![dbuijs](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@dbuijs](https://discuss.elastic.co/u/dbuijs)\ **Post date:** [November 22, 2019, 3:58pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/13 "2019-11-22T15:58:31Z")
It seems like there's significant variability in the value of \_superset\_size, even without touching anything, just running the same query repeatedly. What is concerning is that every time it reports that it's heard back from all shards, and the reported upper bound for error is about 5K, but the actual variation I'm seeing within a few seconds is more than 2M (index is read-only, nothign is getting written). I"ll also add that superset\_size is quite a bit bigger than the total document count in the index. I'm guessing that this is a result of the fields having arrays with more than one value, so it doesn't worry me as much as the number changing. Is this normal behaviour? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 22, 2019, 4:06pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/14 "2019-11-22T16:06:57Z")
Do you have updates to docs or deletes? Do you have replicas? There are [documented issues](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html#_approximate_counts) with how stats are calculated with regards to deletes which may vary between replicas due to differing merges. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 20, 2019, 4:07pm UTC](https://discuss.elastic.co/t/significant-terms-aggregation-custom-score-with-more-than-one-background/208910/15 "2019-12-20T16:07:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.