# How to calculate similarity between files, while i just index their url and metadata **URL:** \ **Category:** Elasticsearch\ **Created:** [July 10, 2019, 11:02am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728 "2019-07-10T11:02:26Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![FeizNouri](https://avatars.discourse-cdn.com/v4/letter/f/7993a0/32.png) [@FeizNouri](https://discuss.elastic.co/u/FeizNouri)\ **Post date:** [July 10, 2019, 11:02am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/1 "2019-07-10T11:02:27Z")
Hi, I'm using FSCrawler to index files to ES, nice solution BTW, but I don't want to index the content so I just index metadata and the URL, so far so good, and my final goal is to calculate the similarity between these files with their content but without indexing it (the user will add new file and we will give him some files that are similar with scores). can I make the comparison by reading the files directly through the URL and compare it by the content? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 10, 2019, 11:03am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/2 "2019-07-10T11:03:31Z")
Nope, you'd need to do that in code external to Elasticsearch. ---
**Author:** ![FeizNouri](https://avatars.discourse-cdn.com/v4/letter/f/7993a0/32.png) [@FeizNouri](https://discuss.elastic.co/u/FeizNouri)\ **Post date:** [July 10, 2019, 11:08am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/3 "2019-07-10T11:08:55Z")
can you give me some idea plz, you mean i can indeed index the url and metadata in ES and when it comes to similarity i can't happen in ES but it can be done in external code? and in external code what libraries or projects that can help me like Tika or idk. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 10, 2019, 11:09am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/4 "2019-07-10T11:09:51Z")
Depends what you mean by similarity I guess? ---
**Author:** ![FeizNouri](https://avatars.discourse-cdn.com/v4/letter/f/7993a0/32.png) [@FeizNouri](https://discuss.elastic.co/u/FeizNouri)\ **Post date:** [July 10, 2019, 11:14am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/5 "2019-07-10T11:14:56Z")
the user will add his file and i need to give him files that are similar to his ( simply similar in the content nothing complex) also every file will have some score for similatrity. i guess same meaning for similarity in ES. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 10, 2019, 11:15am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/6 "2019-07-10T11:15:42Z")
Ok, so how will you calculate that? ---
**Author:** ![FeizNouri](https://avatars.discourse-cdn.com/v4/letter/f/7993a0/32.png) [@FeizNouri](https://discuss.elastic.co/u/FeizNouri)\ **Post date:** [July 10, 2019, 11:30am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/7 "2019-07-10T11:30:38Z")
i'm asking for ideas to help me, because i can do that in ES but by indexing the content of the files, and when i give a certain file it returns files that are similar with a score. i'm asking how i can calculate similarity but without indexing the content. ---
**Author:** ![rvanegmond](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@rvanegmond](https://discuss.elastic.co/u/rvanegmond)\ **Post date:** [July 10, 2019, 11:49am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/8 "2019-07-10T11:49:06Z")
If I'm guessing right he want's to calculate a simularity based on one or more of the metadata fields. Let's say filename and author. I can think of something like: "CV of Jeroen.docx" and "Copy of cv of Jeroen.docx". Both with author "Jeroen". ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 10, 2019, 11:49am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/9 "2019-07-10T11:49:39Z")
> [@FeizNouri](#): > > i'm asking how i can calculate similarity but without indexing the content. It appears to be the content, not the metadata. ---
**Author:** ![FeizNouri](https://avatars.discourse-cdn.com/v4/letter/f/7993a0/32.png) [@FeizNouri](https://discuss.elastic.co/u/FeizNouri)\ **Post date:** [July 10, 2019, 11:54am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/10 "2019-07-10T11:54:26Z")
@rvanegmond no not based on metadata but on content, which is not indexed in ES, I know it is not logic, but the reason why I think of that is that I tried to index the content in ES and calculate similarity and it works but I have so many large files and it doesn't seem logical to index all of their content, I'm looking for a way to be able to calculate similarity with just indexing URL and metadata and maybe at the time of calculating, it reads the files. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 10, 2019, 2:23pm UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/11 "2019-07-10T14:23:14Z")
> [@FeizNouri](#): > > I'm using FSCrawler to index files to ES, nice solution BTW Thanks! 😊 About the question, I wonder if you could use this [ingest-anonymize plugin](https://github.com/sektorcap/ingest-anonymize) to compute a fingerprint at index time which will help to identify similar documents may be? FSCrawler can apply an ingest pipeline as explained in [Elasticsearch settings — FSCrawler 2.10-SNAPSHOT documentation](https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#ingest-node) May be this could help? Note that this requires to index the document. Another solution could be to [start FSCrawler as a REST Service](https://fscrawler.readthedocs.io/en/latest/admin/fs/rest.html) and use the [simulate](https://fscrawler.readthedocs.io/en/latest/admin/fs/rest.html#simulate-upload) endpoint. Once you get back the JSON content (which is not indexed then), you can compute a fingerprint using your code or an ingest pipeline again and use the [simulate ingest API](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html). Only some thoughts here. Not sure it helps. ---
**Author:** ![FeizNouri](https://avatars.discourse-cdn.com/v4/letter/f/7993a0/32.png) [@FeizNouri](https://discuss.elastic.co/u/FeizNouri)\ **Post date:** [July 10, 2019, 3:19pm UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/12 "2019-07-10T15:19:02Z")
GREAT thoughts ! i will explore them and try to solve my problem here and will get back to you for feedback or questions ---
**Author:** ![FeizNouri](https://avatars.discourse-cdn.com/v4/letter/f/7993a0/32.png) [@FeizNouri](https://discuss.elastic.co/u/FeizNouri)\ **Post date:** [July 17, 2019, 8:11am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/13 "2019-07-17T08:11:26Z")
Hi, unfortunately, I could not use what you have suggested because I didn't find a way to compute similarity with the fingerprints or hashing. but I think I need a simple solution so now I'm just trying to compute similarities between files ( or just given a word a get the most relevant documents, like a search engine) with just the metadata, that I index in ES, of the documents. my question here is there a way to make sure that the metadata holds always a good and enough info about the document? what do you think about the plugin carrot2 can it help ? [https://github.com/carrot2/elasticsearch-carrot2](https://github.com/carrot2/elasticsearch-carrot2) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 17, 2019, 8:38am UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/14 "2019-07-17T08:38:33Z")
> [@FeizNouri](#): > > what do you think about the plugin carrot2 can it help ? I don't know it. > [@FeizNouri](#): > > my question here is there a way to make sure that the metadata holds always a good and enough info about the document? Do you mean "can I be guaranteed that FSCrawler / Tika will always extract metadata?" I guess it depends on your source documents but unless there is a bug I don't think that Tika will extract less data. ---
**Author:** ![FeizNouri](https://avatars.discourse-cdn.com/v4/letter/f/7993a0/32.png) [@FeizNouri](https://discuss.elastic.co/u/FeizNouri)\ **Post date:** [July 17, 2019, 2:13pm UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/15 "2019-07-17T14:13:51Z")
ok thank you I think it will works ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 14, 2019, 2:13pm UTC](https://discuss.elastic.co/t/how-to-calculate-similarity-between-files-while-i-just-index-their-url-and-metadata/189728/16 "2019-08-14T14:13:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.