# Creating a MDM **URL:** \ **Category:** Elasticsearch\ **Created:** [June 25, 2016, 12:30am UTC](https://discuss.elastic.co/t/creating-a-mdm/53929 "2016-06-25T00:30:43Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![David\_she](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_she/32/48792_2.png) [@David\_she](https://discuss.elastic.co/u/David_she)\ **Post date:** [June 25, 2016, 12:30am UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/1 "2016-06-25T00:30:43Z")
Hi, We are using elasticsearch and have around 500,000,000 person records. I'm looking a creating a MDM on persons. I'm thinking Java but worried about performance. I'm happy to get my head around Spark or use Mapreduce. Any tips or has anyone else done a similar project. Appreciate your help. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 25, 2016, 8:50pm UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/2 "2016-06-25T20:50:56Z")
What's MDM? ---
**Author:** ![David\_she](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_she/32/48792_2.png) [@David\_she](https://discuss.elastic.co/u/David_she)\ **Post date:** [June 26, 2016, 3:55am UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/3 "2016-06-26T03:55:13Z")
A Master Data Management system. Basically doing entity resolution on a person. E.g. if the name is John Smith is it the same John Smith as another one in the corpus? Based on DOB or passport, address or some other metadata field. Thanks ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 26, 2016, 6:48am UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/4 "2016-06-26T06:48:30Z")
Ahh ok. Graph can totally do that for you! Check out [https://www.elastic.co/products/graph](https://www.elastic.co/products/graph) or [https://www.elastic.co/guide/en/graph/current/index.html](https://www.elastic.co/guide/en/graph/current/index.html) There are also a bunch of great blog posts that explain it. ---
**Author:** ![David\_she](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_she/32/48792_2.png) [@David\_she](https://discuss.elastic.co/u/David_she)\ **Post date:** [June 26, 2016, 9:05am UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/5 "2016-06-26T09:05:56Z")
Champion I'll have a look ---
**Author:** ![ryan\_3\_thomas](https://avatars.discourse-cdn.com/v4/letter/r/7ba0ec/32.png) [@ryan\_3\_thomas](https://discuss.elastic.co/u/ryan_3_thomas)\ **Post date:** [February 2, 2017, 10:38am UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/6 "2017-02-02T10:38:32Z")
Hi David, I'm looking to implement something similar. I'd be grateful for your experience on this project and any pitfalls to avoid. Thanks, Ryan ---
**Author:** ![David\_she](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_she/32/48792_2.png) [@David\_she](https://discuss.elastic.co/u/David_she)\ **Post date:** [February 2, 2017, 7:45pm UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/7 "2017-02-02T19:45:41Z")
Hi Ryan We have just started using Graph. The issue is scale because we then need a second pass using mapreduce and string comparators. We tried Scala Spark but the recursive nature of comparison killed us. I'll let you know more in a couple of weeks. Appreciate any feedback on your findings ---
**Author:** ![ryan\_3\_thomas](https://avatars.discourse-cdn.com/v4/letter/r/7ba0ec/32.png) [@ryan\_3\_thomas](https://discuss.elastic.co/u/ryan_3_thomas)\ **Post date:** [February 3, 2017, 3:41pm UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/8 "2017-02-03T15:41:07Z")
HI David, Have seen similar things on other threads. One of the things I have had recommended to me is to have a dedicated graph database e.g. neo4j and essentially do an enrichment between elastic and neo4j to give the final score and threshold for the MDM key. It requires 2 stores of the same data and a custom interface between the 2 but seems to get around the scale issue. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 3, 2017, 5:28pm UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/9 "2017-02-03T17:28:52Z")
Curious - do you need pre-fused entities or can you assemble them as-you-go? It's the difference between working with pre-fused entities like this: ``` { "fused_entity_id": 435223 "keys": ["passport1", "handle1", "handle2" ...] } ``` (which take a lot of maintenance using _any_ tech stack) and ... ``` { "doc_id": 1, "keys": ["passport1", "handle1"] } { "doc_id": 2, "keys": ["passport1", "handle2"] } { "doc_id": 3, "keys": ["unrelated_passport", "unrelated handle"] } ``` ... where a single entity can be assembled on-the-fly using the graph API to walk the chain of identifiers in your entity sightings data. The advantage of the latter approach is a user can experiment with fuzzier keys that would over-link if you tried using them in your batch linking rules that maintain all entities at index-time. ---
**Author:** ![David\_she](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_she/32/48792_2.png) [@David\_she](https://discuss.elastic.co/u/David_she)\ **Post date:** [February 4, 2017, 8:53am UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/10 "2017-02-04T08:53:06Z")
On the fly is perfect. Love to see an example on github to get me started ---
**Author:** ![David\_she](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_she/32/48792_2.png) [@David\_she](https://discuss.elastic.co/u/David_she)\ **Post date:** [February 4, 2017, 9:46am UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/11 "2017-02-04T09:46:50Z")
Hey Mark Just to let you know it has finally clicked, I knew Graph Theory was nice but REALLY nice embedded in ElasticSearch. Very excited about what can be achieved with Graph and ElasticSearch. Building some demos now. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 6, 2017, 8:58am UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/12 "2017-02-06T08:58:40Z")
Good to know. The trick is to ensure that each entity sighting has its own dedicated document - I call these an "entity reference". Original business documents like an insurance claim may contain references to several different people entities (driver, third party, witness...). With multi-entity docs like this each of these people should have their keys broken out into different `person_reference` child docs to avoid muddling the different identities. Each `person_reference` doc contains: - several different keys that can be used to identify that person (name+DOB+zipcode, SSN, email address...) - the id of the source business document e.g. our insurance claim number This gives you all you need to crawl the graph of keys that make up the aliases for a person entity which you can then visually group in the Graph UI as a single vertex (I'm thinking of adding a button to help automate this UI grouping). From each person entity you can also crawl out via the IDs of original business docs to discover connections to other entities (people, cars, addresses). I plan to cover these techniques in more depth in a future blog or talk. Cheers Mark ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:03pm UTC](https://discuss.elastic.co/t/creating-a-mdm/53929/13 "2017-07-05T22:03:32Z")