# Elasticsearch get all unique Ids

**URL:** <https://discuss.elastic.co/t/elasticsearch-get-all-unique-ids/292589>\
**Category:** Elasticsearch\
**Created:** [December 21, 2021, 5:45pm UTC](https://discuss.elastic.co/t/elasticsearch-get-all-unique-ids/292589 "2021-12-21T17:45:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![iamtheschmitzer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamtheschmitzer/32/96988_2.png) [@iamtheschmitzer](https://discuss.elastic.co/u/iamtheschmitzer)\
**Post date:** [December 21, 2021, 5:45pm UTC](https://discuss.elastic.co/t/elasticsearch-get-all-unique-ids/292589/1 "2021-12-21T17:45:37Z")

</div>

Thanks to all who read this for your help.

Background: I have a job that when executing inserts a document. The job can fail and run again, inserting a second document. Both have the same linkId- so I know there is only one job. I need the count of ALL unique linkIds - no matter how many there are.

I first looked at `cardinality` but that has a limit of 40,000 (precision\_threshold) after which it becomes inaccurate.

Then I looked into `cumulative_cardinality` but that strategy uses `cardinality`. But the documentation makes no mention of inaccuracy. Is this method accurate for large numbers?

My aggs section looks like this:

```auto
  "aggs": {
    "unique_attempts": {
      "date_histogram": {
        "field": "timestamp",
        "calendar_interval": "day"
      },
      "aggs": {
        "distinct_attempts": {
          "cardinality": {
            "field": "linkId"
          }
        },
        "total_attempts": {
          "cumulative_cardinality": {
            "buckets_path": "distinct_attempts"
          }
        }
      }
    }
  }

```

So is this an approach that will give me an accurate count of unique `localId`? Or do I make multiple calls using the API and join the results myself? If API - what strategy would be best?

Many thanks this is not an easy question

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 21, 2021, 7:06pm UTC](https://discuss.elastic.co/t/elasticsearch-get-all-unique-ids/292589/2 "2021-12-21T19:06:06Z")

</div>

Have you considered using a [transform](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/transform-apis.html) to create a separate index with a single document per `localId`? This would allow you to query, filter and count the number of documents, which would give exact values. The trade-off is that it may lag a little.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 21, 2021, 7:11pm UTC](https://discuss.elastic.co/t/elasticsearch-get-all-unique-ids/292589/3 "2021-12-21T19:11:25Z")

</div>

@iamtheschmitzer

@Christian_Dahlqvist provided and excellent advice.

The Latest transform could be a great solution for this...added benefit the last document details are also saved

---

<div class="post-metadata">

**Author:** ![iamtheschmitzer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamtheschmitzer/32/96988_2.png) [@iamtheschmitzer](https://discuss.elastic.co/u/iamtheschmitzer)\
**Post date:** [December 21, 2021, 7:33pm UTC](https://discuss.elastic.co/t/elasticsearch-get-all-unique-ids/292589/4 "2021-12-21T19:33:29Z")

</div>

No, first I've heard of it. I'll take a look. Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 21, 2021, 8:25pm UTC](https://discuss.elastic.co/t/elasticsearch-get-all-unique-ids/292589/5 "2021-12-21T20:25:08Z")

</div>

It's Elasticsearch, not `ElasticSearch` by the way 🙂

---

<div class="post-metadata">

**Author:** ![iamtheschmitzer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamtheschmitzer/32/96988_2.png) [@iamtheschmitzer](https://discuss.elastic.co/u/iamtheschmitzer)\
**Post date:** [December 21, 2021, 9:08pm UTC](https://discuss.elastic.co/t/elasticsearch-get-all-unique-ids/292589/6 "2021-12-21T21:08:12Z")

</div>

👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 18, 2022, 9:08pm UTC](https://discuss.elastic.co/t/elasticsearch-get-all-unique-ids/292589/7 "2022-01-18T21:08:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.