# Clustering data on Elasticsearch index **URL:** \ **Category:** Elasticsearch\ **Created:** [June 20, 2016, 4:33am UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262 "2016-06-20T04:33:55Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![nfantone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfantone/32/10404_2.png) [@nfantone](https://discuss.elastic.co/u/nfantone)\ **Post date:** [June 20, 2016, 4:33am UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/1 "2016-06-20T04:33:55Z")
Hi, everyone! I created an account here because having asked a question over at SO, received absolutely no love (I even got the _Thumbleweed_ badge for it - no kidding). Here it is: [http://stackoverflow.com/questions/37734131/clustering-data-in-elasticsearch](http://stackoverflow.com/questions/37734131/clustering-data-in-elasticsearch) Could someone give me a hand on this? It'd be much appreciated. **EDIT** As requested, I'm inlining my original Stackoverflow question over here. Find its content below: > I have a rather large set of customer purchases stored in an Elasticsearch index. What I'd like to do is group customers on the set and generate a new index from that data in such a way that'd allow me to: > Differentiate unique customers. > Have aggregated information on each entry (such as sums and avg of a number of other fields). > My problem comes with the business definition I was given for "uniqueness" of customers. Two customers are considered to be same if at least 75% or their properties match (like "country", "language", "email" and so on). Properties are dynamically added during user profile creation and they might change, be added or removed in the future. > This seems closely related to how a terms filter with a minimum\_should\_match of 75% resolves things. So, my question is: is there a way of clustering data in Elasticsearch 2.0+ that would fit my scenario? Ideally, it would behave like a multi-bucket aggregation that would group documents if more than 3/4 of their attributes' values match each other. > EDIT: I'm not looking for manual solutions like iterating each document and query the index to retrieve similar results. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 20, 2016, 4:34am UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/2 "2016-06-20T04:34:57Z")
It'd probably get more interest if you reposted the question too 🙂 ---
**Author:** ![nfantone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfantone/32/10404_2.png) [@nfantone](https://discuss.elastic.co/u/nfantone)\ **Post date:** [June 20, 2016, 4:56am UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/3 "2016-06-20T04:56:44Z")
Well, the whole point was to _not_ do that and avoid polluting the Internet, since it's just one click away. But if you think it'll help, I'll edit my original post to include it. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 20, 2016, 4:57am UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/4 "2016-06-20T04:57:44Z")
> [@nfantone](#): > > Well, the whole point was to not do that and avoid polluting the Internet, since it's just one click away. I think you under estimate the laziness of the internet 😛 ---
**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\ **Post date:** [June 20, 2016, 7:37am UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/5 "2016-06-20T07:37:22Z")
I think a more like this query might be able to help you: [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html) There is a proposal for a fingerprinting inget processor here: [https://github.com/elastic/elasticsearch/issues/16938](https://github.com/elastic/elasticsearch/issues/16938) that you might find interesting as well. Hope this helps, Isabel ---
**Author:** ![nfantone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfantone/32/10404_2.png) [@nfantone](https://discuss.elastic.co/u/nfantone)\ **Post date:** [June 20, 2016, 4:41pm UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/6 "2016-06-20T16:41:52Z")
This could very well be a viable solution. The only drawback I can think of is that this would require to know the actual doc ids and matching fields beforehand (unless the `fields` option accepts wildcards - something not mentioned on the docs). Am I right? What I'd really like is to find clusters of documents _matching each other_, not some seed document. From the MLT doc: > Suppose we wanted to find all documents similar to a given input document. Obviously, the input document itself should be its best match for that type of query. Then, you can also think this as a two step process in which you first find a list of every seed document (the most representative for each type), and then run an MLT query against that list. The fingerprint processos looks promising. Already +1d it. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [June 20, 2016, 9:30pm UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/7 "2016-06-20T21:30:27Z")
If you want to group documents by statistics of field similarity, you need to compute the distance between documents. Each doc field stands for a feature, so your problem is a feature space topology similarity computation. If you want to compute all groups, you must measure the distance of each document to all other documents, by iterating over the docs. The MLT query simplifies such an approach. It can be executed for each doc to build doc groups. The algorithm behind MLT is k-nearest-neighbor. See [https://en.wikipedia.org/wiki/K-nearest\_neighbors\_algorithm](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) Lucene has a builtin k-nearest-neighbor classifier [http://lucene.apache.org/core/5\_5\_0/classification/org/apache/lucene/classification/package-summary.html](http://lucene.apache.org/core/5_5_0/classification/org/apache/lucene/classification/package-summary.html) but to expose that implementation in Elasticsearch is not trivial. ---
**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\ **Post date:** [June 21, 2016, 9:23am UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/8 "2016-06-21T09:23:28Z")
> [@nfantone](#): > > The fingerprint processos looks promising. Already +1d it. [Identifying and Filtering Near-Duplicate Documents | SpringerLink](http://link.springer.com/chapter/10.1007/3-540-45123-4_1) this is one of the algorithms that I believe would be made easier by having fingerprinting support. Isabel ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [June 21, 2016, 11:25am UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/9 "2016-06-21T11:25:20Z")
> [@nfantone](#): > > Two customers are considered to be same if at least 75% or their properties match (like "country", "language", "email" and so on) If I understand the question correctly you are talking about "entity resolution" - identifying each real-world person from a set of documents that may have described the same person in different ways. A key question is are you doing this to: 1. _assist_ users (as in the system suggests "_person A might be person B - please confirm_") or 2. _automate_ some batch data de-duplication process with minimal human intervention. Fuzzy matching rules like MLT are useful for scenario #1 (emphasis being on recall vs precision) but scenario #2 requires more rigour and emphasises precision over recall. Each merge operation has to be something that can be trusted for the algorithm to iterate on without human intervention. A person cannot be allowed to over-link and become his brother then their father etc through a steady accumulation of weakly linked properties. Large-scale iterative entity resolution can be achieved but uses non-fuzzy keys and lots of different ways of composing the keys. I demoed this 37 minutes into this presentation at elasticon: [https://www.elastic.co/elasticon/conf/2016/sf/graph-capabilities-in-the-elastic-stack](https://www.elastic.co/elasticon/conf/2016/sf/graph-capabilities-in-the-elastic-stack) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:41pm UTC](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262/10 "2017-07-05T22:41:46Z")