# Aggregation across multiple indexes/indices - significant terms **URL:** \ **Category:** Elasticsearch\ **Created:** [February 9, 2022, 9:09pm UTC](https://discuss.elastic.co/t/aggregation-across-multiple-indexes-indices-significant-terms/296787 "2022-02-09T21:09:23Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![anthonyfps](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@anthonyfps](https://discuss.elastic.co/u/anthonyfps)\ **Post date:** [February 9, 2022, 9:09pm UTC](https://discuss.elastic.co/t/aggregation-across-multiple-indexes-indices-significant-terms/296787/1 "2022-02-09T21:09:23Z")
Hi, we have indexes that are split by date for manageability but contain the same mappings. I'm currently trying to use the significant terms aggregation to identify new terms, by specifying the foreground as e.g. the last day and the background as the rest of history. When the aggregation is confined to one index it works as expected, however the background frequencies with multiple indexes do not include the same fields across all the indices, I imagine only including the background frequency within the index that the term was found in. Is this expected behaviour? ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [February 12, 2022, 3:39pm UTC](https://discuss.elastic.co/t/aggregation-across-multiple-indexes-indices-significant-terms/296787/2 "2022-02-12T15:39:15Z")
> [@anthonyfps](#): > > the background frequencies with multiple indexes do not include the same fields across all the indices, Could explain more about this? Do background frequencies contain other fields from other indices? > [@anthonyfps](#): > > I imagine only including the background frequency within the index that the term was found in. I Why do you think so? I suppose its reasonable background frequency calculated over all indices regardless of whether the index contains the term. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 13, 2022, 12:37pm UTC](https://discuss.elastic.co/t/aggregation-across-multiple-indexes-indices-significant-terms/296787/3 "2022-02-13T12:37:50Z")
Hi Anthony. Yes, this is expected behaviour. Background frequency checks on potentially millions of candidate terms is expensive and so the implementation works with local stats found in a shard. The goal of finding “what is significant today?” is just not feasible using only day-based indices. Distributed data is the enemy of a lot of analytical functions I’m afraid. ---
**Author:** ![anthonyfps](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@anthonyfps](https://discuss.elastic.co/u/anthonyfps)\ **Post date:** [February 17, 2022, 1:24pm UTC](https://discuss.elastic.co/t/aggregation-across-multiple-indexes-indices-significant-terms/296787/4 "2022-02-17T13:24:20Z")
Thanks for replying! I was imprecise here, I meant that the multiple indexes do not include the same terms rather than fields - it seems that statistics are only collated locally (it seems to a shard) rather than across the whole group of indexes. The bg\_count number seems to include the total across the indices however, up to a limit. I found that the background count did not include occurences of the term in other indices. ---
**Author:** ![anthonyfps](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@anthonyfps](https://discuss.elastic.co/u/anthonyfps)\ **Post date:** [February 17, 2022, 1:25pm UTC](https://discuss.elastic.co/t/aggregation-across-multiple-indexes-indices-significant-terms/296787/5 "2022-02-17T13:25:30Z")
Thanks for getting back to me. Sounds reasonable. The background count for that term seems to be an extra search, it would have to be limited to the local shard or there may be in aggregate many many terms to search for across all the shards. It's straightforward enough to compile the information another way. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 17, 2022, 1:26pm UTC](https://discuss.elastic.co/t/aggregation-across-multiple-indexes-indices-significant-terms/296787/6 "2022-03-17T13:26:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.