# Elasticsearch for Data Science

**URL:** <https://discuss.elastic.co/t/elasticsearch-for-data-science/339818>\
**Category:** Elasticsearch\
**Created:** [August 1, 2023, 3:02pm UTC](https://discuss.elastic.co/t/elasticsearch-for-data-science/339818 "2023-08-01T15:02:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![tmslara.a](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tmslara.a/32/116339_2.png) [@tmslara.a](https://discuss.elastic.co/u/tmslara.a)\
**Post date:** [August 1, 2023, 3:02pm UTC](https://discuss.elastic.co/t/elasticsearch-for-data-science/339818/1 "2023-08-01T15:02:14Z")

</div>

Hi,  
Some context. I'm using Elasticsearch and filebeat to store documents. I have 6 fields. One field represents the timestamp and the other 5 are keywords. Two fields correspond to IDs (id\_1 and id\_2). The IDs have many different values, for example, in a single day the cardinality of id\_1 is around 4 million.

I want to apply machine learning techniques on the stored data. More precisely, in this first iteration I want to cluster the information (for example using k-means). Before trying to apply the algorithm I'm performing an exploration of the dataset. I'm interested on the cardinality of id\_2 per id\_1. To do this, I'm using a terms aggregation along with a cardinality sub aggregation. However, as I said before I have around 4 million id\_1 values and the limit of the terms aggregation is around 65,000 terms. The structure of the aggregation is the following

```auto
"aggs": {
  "terms_id_1": {
    "terms": {
      "field": "id_1",
      "size": 10000
    },
    "aggs": {
      "card_id_2": {
        "cardinality": {
          "field": "id_2"
        }
      },
      "id_2_bucket_sort": {
        "bucket_sort": {
          "sort": [
            {"card_id_2": {"order": "desc"}}
          ]
        }
      }
    }
  }
}

```

I have two questions:

1. Is it possible to perform the aggregation? I've tried before using the partition parameter of the term aggregation but I think it is not suitable to consider the complete period.
2. Is Elasticsearch suitable for its use in the context of Machine Learning (supervised or unsupervised) or I should consider another tool. I've made research about this topic and the opinion are heterogeneous.

Best regards,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 29, 2023, 3:02pm UTC](https://discuss.elastic.co/t/elasticsearch-for-data-science/339818/2 "2023-08-29T15:02:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.