# Finding similar documents with Elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [January 19, 2014, 6:13am UTC](https://discuss.elastic.co/t/finding-similar-documents-with-elasticsearch/15312 "2014-01-19T06:13:53Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Zoran\_Jeremic](https://avatars.discourse-cdn.com/v4/letter/z/7ba0ec/32.png) [@Zoran\_Jeremic](https://discuss.elastic.co/u/Zoran_Jeremic)\ **Post date:** [January 19, 2014, 6:13am UTC](https://discuss.elastic.co/t/finding-similar-documents-with-elasticsearch/15312/1 "2014-01-19T06:13:53Z")
Hi guys, I'm trying to develop service that will store uploaded files as attachment (file is one field in document). This part works fine as I can search these files using like\_text as input. However, the second part of this service should compare the file that is just uploaded with the existing files in order to find duplicates or very similar files. The problem is that I always get the same results regardless the input I'm using, and these results are wrong as exactly the same file has smallest score very often. It looks that like\_text extracted from uploaded file is always the same, and none of the documents has expected score, which should be I believe 1 in case of identical documents. The scores I get are always less then 0.2. Could you please check if there is something wrong with my code? String mapping = copyToStringFromClasspath( "/org/prosolo/services/indexing/documents-mapping.json"); byte[] txt = org.elasticsearch.common.io.Streams.copyToByteArray(file); Client client = ElasticSearchFactory.getClient(); client.admin().indices().putMapping(putMappingRequest(indexName).type(indexType).source(mapping)).actionGet(); IndexResponse iResponse = client.index(indexRequest(indexName).type(indexType) .source(jsonBuilder() .startObject() .field("file", txt) .field("title",title) .field("visibility",visibilityType.name().toLowerCase()) .field("ownerId",ownerId) .field("description",description) .field("contentType",DocumentType.DOCUMENT.name().toLowerCase()) .field("dateCreated",dateCreated) .field("url",link) .field("relatedToType",relatedToType) .field("relatedToId",relatedToId) .endObject())) .actionGet(); client.admin().indices().refresh(refreshRequest()).actionGet(); ``` MoreLikeThisRequestBuilder mltRequestBuilder=new ``` MoreLikeThisRequestBuilder(client, ESIndexNames.INDEX\_DOCUMENTS, ESIndexTypes.DOCUMENT, iResponse.getId()); mltRequestBuilder.setField("file"); SearchResponse response = client.moreLikeThis(mltRequestBuilder.request()).actionGet(); SearchHits searchHits= response.getHits(); System.out.println("getTotalHits:"+searchHits.getTotalHits()); Iterator hitsIter=searchHits.iterator(); while(hitsIter.hasNext()){ SearchHit searchHit=hitsIter.next(); System.out.println("FOUND DOCUMENT:"+searchHit.getId()+" title:"+searchHit.getSource().get("title")+" score:"+searchHit.score()); } And this is the mapping I was using { "document":{ "properties":{ "title":{ "type":"string", "store":true }, "description":{ "type":"string", "store":"yes" }, "contentType":{ "type":"string", "store":"yes" }, "dateCreated":{ "store":"yes", "type":"date" }, "url":{ "store":"yes", "type":"string" }, "visibility": { "store":"yes", "type":"string" }, "ownerId": { "type": "long", "store":"yes" }, "relatedToType": { "type": "string", "store":"yes" }, "relatedToId": { "type": "long", "store":"yes" }, "file":{ "path": "full", "type":"attachment", "fields":{ "author": { "type": "string" }, "title": { "store": true, "type": "string" }, "keywords": { "type": "string" }, "file": { "store": true, "term\_vector": "with\_positions\_offsets", "type": "string" }, "name": { "type": "string" }, "content\_length": { "type": "integer" }, "date": { "format": "dateOptionalTime", "type": "date" }, "content\_type": { "type": "string" } } } } } } Thanks, Zoran -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/79e29c89-62ea-42f3-be93-3e215a75860a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/79e29c89-62ea-42f3-be93-3e215a75860a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 19, 2014, 7:42am UTC](https://discuss.elastic.co/t/finding-similar-documents-with-elasticsearch/15312/2 "2014-01-19T07:42:39Z")
Do you get same results if you compare against file.file field? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 19 janv. 2014 à 07:13, Zoran Jeremic [zoran.jeremic@gmail.com](mailto:zoran.jeremic@gmail.com) a écrit : Hi guys, I'm trying to develop service that will store uploaded files as attachment (file is one field in document). This part works fine as I can search these files using like\_text as input. However, the second part of this service should compare the file that is just uploaded with the existing files in order to find duplicates or very similar files. The problem is that I always get the same results regardless the input I'm using, and these results are wrong as exactly the same file has smallest score very often. It looks that like\_text extracted from uploaded file is always the same, and none of the documents has expected score, which should be I believe 1 in case of identical documents. The scores I get are always less then 0.2. Could you please check if there is something wrong with my code? String mapping = copyToStringFromClasspath("/org/prosolo/services/indexing/documents-mapping.json"); byte[] txt = org.elasticsearch.common.io.Streams.copyToByteArray(file); Client client = ElasticSearchFactory.getClient(); client.admin().indices().putMapping(putMappingRequest(indexName).type(indexType).source(mapping)).actionGet(); IndexResponse iResponse = client.index(indexRequest(indexName).type(indexType) .source(jsonBuilder() .startObject() .field("file", txt) .field("title",title) .field("visibility",visibilityType.name().toLowerCase()) .field("ownerId",ownerId) .field("description",description) .field("contentType",DocumentType.DOCUMENT.name().toLowerCase()) .field("dateCreated",dateCreated) .field("url",link) .field("relatedToType",relatedToType) .field("relatedToId",relatedToId) .endObject())) .actionGet(); client.admin().indices().refresh(refreshRequest()).actionGet(); ``` MoreLikeThisRequestBuilder mltRequestBuilder=new MoreLikeThisRequestBuilder(client, ESIndexNames.INDEX_DOCUMENTS, ESIndexTypes.DOCUMENT, iResponse.getId()); mltRequestBuilder.setField("file"); SearchResponse response = client.moreLikeThis(mltRequestBuilder.request()).actionGet(); SearchHits searchHits= response.getHits(); System.out.println("getTotalHits:"+searchHits.getTotalHits()); Iterator hitsIter=searchHits.iterator(); while(hitsIter.hasNext()){ SearchHit searchHit=hitsIter.next(); System.out.println("FOUND DOCUMENT:"+searchHit.getId()+" title:"+searchHit.getSource().get("title")+" score:"+searchHit.score()); } ``` And this is the mapping I was using { "document":{ "properties":{ "title":{ "type":"string", "store":true }, "description":{ "type":"string", "store":"yes" }, "contentType":{ "type":"string", "store":"yes" }, "dateCreated":{ "store":"yes", "type":"date" }, "url":{ "store":"yes", "type":"string" }, "visibility": { "store":"yes", "type":"string" }, "ownerId": { "type": "long", "store":"yes" }, "relatedToType": { "type": "string", "store":"yes" }, "relatedToId": { "type": "long", "store":"yes" }, "file":{ "path": "full", "type":"attachment", "fields":{ "author": { "type": "string" }, "title": { "store": true, "type": "string" }, "keywords": { "type": "string" }, "file": { "store": true, "term\_vector": "with\_positions\_offsets", "type": "string" }, "name": { "type": "string" }, "content\_length": { "type": "integer" }, "date": { "format": "dateOptionalTime", "type": "date" }, "content\_type": { "type": "string" } ``` } } } } ``` } Thanks, Zoran -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/79e29c89-62ea-42f3-be93-3e215a75860a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/79e29c89-62ea-42f3-be93-3e215a75860a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/E60D0A89-10B2-46C1-83F4-5F18191EBB55%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/E60D0A89-10B2-46C1-83F4-5F18191EBB55%40pilato.fr). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Zoran\_Jeremic](https://avatars.discourse-cdn.com/v4/letter/z/7ba0ec/32.png) [@Zoran\_Jeremic](https://discuss.elastic.co/u/Zoran_Jeremic)\ **Post date:** [January 19, 2014, 10:30pm UTC](https://discuss.elastic.co/t/finding-similar-documents-with-elasticsearch/15312/3 "2014-01-19T22:30:32Z")
Hi David, When I try to compare against file.file, I got exception: Exception in thread "Thread-62" org.elasticsearch.ElasticSearchException: No fields found to fetch the 'likeText' from at org.elasticsearch.action.mlt.TransportMoreLikeThisAction$1.onResponse(TransportMoreLikeThisAction.java:176) at org.elasticsearch.action.mlt.TransportMoreLikeThisAction$1.onResponse(TransportMoreLikeThisAction.java:128) at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:93) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Zoran On Saturday, 18 January 2014 23:42:39 UTC-8, David Pilato wrote: > Do you get same results if you compare against file.file field? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 19 janv. 2014 à 07:13, Zoran Jeremic \<[zoran....@gmail.com](mailto:zoran....@gmail.com) \\> > a écrit : > > Hi guys, > > I'm trying to develop service that will store uploaded files as attachment > (file is one field in document). This part works fine as I can search these > files using like\_text as input. However, the second part of this service > should compare the file that is just uploaded with the existing files in > order to find duplicates or very similar files. The problem is that I > always get the same results regardless the input I'm using, and these > results are wrong as exactly the same file has smallest score very often. > It looks that like\_text extracted from uploaded file is always the same, > and none of the documents has expected score, which should be I believe 1 > in case of identical documents. The scores I get are always less then 0.2. > Could you please check if there is something wrong with my code? > > String mapping = copyToStringFromClasspath( > "/org/prosolo/services/indexing/documents-mapping.json"); > byte txt = org.elasticsearch.common.io.Streams.copyToByteArray(file); > Client client = ElasticSearchFactory.getClient(); > > client.admin().indices().putMapping(putMappingRequest(indexName).type(indexType).source(mapping)).actionGet(); > IndexResponse iResponse = > client.index(indexRequest(indexName).type(indexType) > .source(jsonBuilder() > .startObject() > .field("file", txt) > .field("title",title) > > .field("visibility",visibilityType.name().toLowerCase()) > .field("ownerId",ownerId) > .field("description",description) > .field("contentType",DocumentType.DOCUMENT.name > ().toLowerCase()) > .field("dateCreated",dateCreated) > .field("url",link) > .field("relatedToType",relatedToType) > .field("relatedToId",relatedToId) > .endObject())) > .actionGet(); > client.admin().indices().refresh(refreshRequest()).actionGet(); > > ``` > MoreLikeThisRequestBuilder mltRequestBuilder=new > > ``` > > MoreLikeThisRequestBuilder(client, ESIndexNames.INDEX\_DOCUMENTS, > ESIndexTypes.DOCUMENT, iResponse.getId()); > mltRequestBuilder.setField("file"); > SearchResponse response = > client.moreLikeThis(mltRequestBuilder.request()).actionGet(); > SearchHits searchHits= response.getHits(); > System.out.println("getTotalHits:"+searchHits.getTotalHits()); > Iterator hitsIter=searchHits.iterator(); > while(hitsIter.hasNext()){ > SearchHit searchHit=hitsIter.next(); > System.out.println("FOUND DOCUMENT:"+searchHit.getId()+" > title:"+searchHit.getSource().get("title")+" score:"+searchHit.score()); > } > > And this is the mapping I was using > > { > "document":{ > "properties":{ > "title":{ > "type":"string", > "store":true > }, > "description":{ > "type":"string", > "store":"yes" > }, > "contentType":{ > "type":"string", > "store":"yes" > }, > "dateCreated":{ > "store":"yes", > "type":"date" > }, > "url":{ > "store":"yes", > "type":"string" > }, > "visibility": { > "store":"yes", > "type":"string" > }, > "ownerId": { > "type": "long", > "store":"yes" > }, > "relatedToType": { > "type": "string", > "store":"yes" > }, > "relatedToId": { > "type": "long", > "store":"yes" > }, > "file":{ > "path": "full", > "type":"attachment", > "fields":{ > "author": { > "type": "string" > }, > "title": { > "store": true, > "type": "string" > }, > "keywords": { > "type": "string" > }, > "file": { > "store": true, > "term\_vector": "with\_positions\_offsets", > "type": "string" > }, > "name": { > "type": "string" > }, > "content\_length": { > "type": "integer" > }, > "date": { > "format": "dateOptionalTime", > "type": "date" > }, > "content\_type": { > "type": "string" > } > } > } > } > > } > } > > Thanks, > Zoran > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/79e29c89-62ea-42f3-be93-3e215a75860a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/79e29c89-62ea-42f3-be93-3e215a75860a%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e9fdcee1-37c3-4a0a-a474-c84d40c22393%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e9fdcee1-37c3-4a0a-a474-c84d40c22393%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Zoran\_Jeremic](https://avatars.discourse-cdn.com/v4/letter/z/7ba0ec/32.png) [@Zoran\_Jeremic](https://discuss.elastic.co/u/Zoran_Jeremic)\ **Post date:** [January 22, 2014, 4:31am UTC](https://discuss.elastic.co/t/finding-similar-documents-with-elasticsearch/15312/4 "2014-01-22T04:31:07Z")
Does anyone have an idea what could be the problem here? On Sunday, 19 January 2014 14:30:32 UTC-8, Zoran Jeremic wrote: > Hi David, > > When I try to compare against file.file, I got exception: > > Exception in thread "Thread-62" org.elasticsearch.ElasticSearchException: > No fields found to fetch the 'likeText' from > at > org.elasticsearch.action.mlt.TransportMoreLikeThisAction$1.onResponse(TransportMoreLikeThisAction.java:176) > at > org.elasticsearch.action.mlt.TransportMoreLikeThisAction$1.onResponse(TransportMoreLikeThisAction.java:128) > at > org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:93) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > Zoran > > On Saturday, 18 January 2014 23:42:39 UTC-8, David Pilato wrote: > > > Do you get same results if you compare against file.file field? > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 19 janv. 2014 à 07:13, Zoran Jeremic [zoran....@gmail.com](mailto:zoran....@gmail.com) a écrit : > > > > Hi guys, > > > > I'm trying to develop service that will store uploaded files as > > attachment (file is one field in document). This part works fine as I can > > search these files using like\_text as input. However, the second part of > > this service should compare the file that is just uploaded with the > > existing files in order to find duplicates or very similar files. The > > problem is that I always get the same results regardless the input I'm > > using, and these results are wrong as exactly the same file has smallest > > score very often. It looks that like\_text extracted from uploaded file is > > always the same, and none of the documents has expected score, which should > > be I believe 1 in case of identical documents. The scores I get are always > > less then 0.2. > > Could you please check if there is something wrong with my code? > > > > String mapping = copyToStringFromClasspath( > > "/org/prosolo/services/indexing/documents-mapping.json"); > > byte txt = org.elasticsearch.common.io.Streams.copyToByteArray(file); > > Client client = ElasticSearchFactory.getClient(); > > > > client.admin().indices().putMapping(putMappingRequest(indexName).type(indexType).source(mapping)).actionGet(); > > IndexResponse iResponse = > > client.index(indexRequest(indexName).type(indexType) > > .source(jsonBuilder() > > .startObject() > > .field("file", txt) > > .field("title",title) > > > > .field("visibility",visibilityType.name().toLowerCase()) > > .field("ownerId",ownerId) > > .field("description",description) > > .field("contentType",DocumentType.DOCUMENT.name > > ().toLowerCase()) > > .field("dateCreated",dateCreated) > > .field("url",link) > > .field("relatedToType",relatedToType) > > .field("relatedToId",relatedToId) > > .endObject())) > > .actionGet(); > > client.admin().indices().refresh(refreshRequest()).actionGet(); > > > > ``` > > MoreLikeThisRequestBuilder mltRequestBuilder=new > > > > ``` > > > > MoreLikeThisRequestBuilder(client, ESIndexNames.INDEX\_DOCUMENTS, > > ESIndexTypes.DOCUMENT, iResponse.getId()); > > mltRequestBuilder.setField("file"); > > SearchResponse response = > > client.moreLikeThis(mltRequestBuilder.request()).actionGet(); > > SearchHits searchHits= response.getHits(); > > System.out.println("getTotalHits:"+searchHits.getTotalHits()); > > Iterator hitsIter=searchHits.iterator(); > > while(hitsIter.hasNext()){ > > SearchHit searchHit=hitsIter.next(); > > System.out.println("FOUND DOCUMENT:"+searchHit.getId()+" > > title:"+searchHit.getSource().get("title")+" score:"+searchHit.score()); > > } > > > > And this is the mapping I was using > > > > { > > "document":{ > > "properties":{ > > "title":{ > > "type":"string", > > "store":true > > }, > > "description":{ > > "type":"string", > > "store":"yes" > > }, > > "contentType":{ > > "type":"string", > > "store":"yes" > > }, > > "dateCreated":{ > > "store":"yes", > > "type":"date" > > }, > > "url":{ > > "store":"yes", > > "type":"string" > > }, > > "visibility": { > > "store":"yes", > > "type":"string" > > }, > > "ownerId": { > > "type": "long", > > "store":"yes" > > }, > > "relatedToType": { > > "type": "string", > > "store":"yes" > > }, > > "relatedToId": { > > "type": "long", > > "store":"yes" > > }, > > "file":{ > > "path": "full", > > "type":"attachment", > > "fields":{ > > "author": { > > "type": "string" > > }, > > "title": { > > "store": true, > > "type": "string" > > }, > > "keywords": { > > "type": "string" > > }, > > "file": { > > "store": true, > > "term\_vector": "with\_positions\_offsets", > > "type": "string" > > }, > > "name": { > > "type": "string" > > }, > > "content\_length": { > > "type": "integer" > > }, > > "date": { > > "format": "dateOptionalTime", > > "type": "date" > > }, > > "content\_type": { > > "type": "string" > > } > > } > > } > > } > > > > } > > } > > > > Thanks, > > Zoran > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/79e29c89-62ea-42f3-be93-3e215a75860a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/79e29c89-62ea-42f3-be93-3e215a75860a%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/069f14b7-417c-4e0c-b87b-3ab5c60a7600%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/069f14b7-417c-4e0c-b87b-3ab5c60a7600%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:55am UTC](https://discuss.elastic.co/t/finding-similar-documents-with-elasticsearch/15312/5 "2017-07-06T01:55:27Z")