# Automatic Keywords extraction in ElasticSearch **URL:** \ **Category:** Elasticsearch\ **Created:** [February 16, 2015, 1:00pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187 "2015-02-16T13:00:38Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Marria](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@Marria](https://discuss.elastic.co/u/Marria)\ **Post date:** [February 16, 2015, 1:00pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/1 "2015-02-16T13:00:38Z")
Hi all, I started using ElasticSearch to index my corpus of PDF files, I succeeded in indexing my PDF files as attachments (base64), my search queries on the content go right but I couldn't find how to extract automaticaly keywords from these files in ElasticSearch. Is it possible to do that with ElasticSearch or not? Could anybody help with relevent links or advices?? Thanks a lot. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 16, 2015, 1:10pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/2 "2015-02-16T13:10:09Z")
Are you using mapper attachment plugin or do you extract text yourself using Tika or similar ? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 16 févr. 2015 à 14:00, Marria [m\_bekrar@esi.dz](mailto:m_bekrar@esi.dz) a écrit : > > Hi all, > > I started using Elasticsearch to index my corpus of PDF files, I succeeded in indexing my PDF files as attachments (base64), my search queries on the content go right but I couldn't find how to extract automaticaly keywords from these files in Elasticsearch. Is it possible to do that with Elasticsearch or not? > > Could anybody help with relevent links or advices?? > > Thanks a lot. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6272F6F7-2540-4527-B4CF-E7503CD44E9F%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/6272F6F7-2540-4527-B4CF-E7503CD44E9F%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marria](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@Marria](https://discuss.elastic.co/u/Marria)\ **Post date:** [February 16, 2015, 1:22pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/3 "2015-02-16T13:22:08Z")
Hi David, thank you for your answer. I am using mapper attachment plugin. Le lundi 16 février 2015 14:10:25 UTC+1, David Pilato a écrit : > Are you using mapper attachment plugin or do you extract text yourself > using Tika or similar ? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 16 févr. 2015 à 14:00, Marria \\> a écrit : > > Hi all, > > I started using Elasticsearch to index my corpus of PDF files, I succeeded > in indexing my PDF files as attachments (base64), my search queries on the > content go right but I couldn't find how to extract automaticaly keywords > from these files in Elasticsearch. Is it possible to do that with > Elasticsearch or not? > > Could anybody help with relevent links or advices?? > > Thanks a lot. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d0a4c1e8-fffc-4a7e-9f22-670b8e3a4618%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d0a4c1e8-fffc-4a7e-9f22-670b8e3a4618%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 16, 2015, 1:29pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/4 "2015-02-16T13:29:25Z")
So the mapper attachment plugin tries to extract keywords: [https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java#L533-542](https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java#L533-542) [https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java#L533-542](https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java#L533-542) Why do you think they are not extracted? Note that mapper plugin never modifies \_source which contains exactly what you sent to elasticsearch. -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > Le 16 févr. 2015 à 14:22, Marria [m\_bekrar@esi.dz](mailto:m_bekrar@esi.dz) a écrit : > > Hi David, > > thank you for your answer. > > I am using mapper attachment plugin. > > Le lundi 16 février 2015 14:10:25 UTC+1, David Pilato a écrit : > Are you using mapper attachment plugin or do you extract text yourself using Tika or similar ? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 16 févr. 2015 à 14:00, Marria \\> a écrit : > > > Hi all, > > > > I started using Elasticsearch to index my corpus of PDF files, I succeeded in indexing my PDF files as attachments (base64), my search queries on the content go right but I couldn't find how to extract automaticaly keywords from these files in Elasticsearch. Is it possible to do that with Elasticsearch or not? > > > > Could anybody help with relevent links or advices?? > > > > Thanks a lot. > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/fab9d7ba-502b-4166-874a-552834be18db%40googlegroups.com?utm_medium=email&utm_source=footer). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d0a4c1e8-fffc-4a7e-9f22-670b8e3a4618%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d0a4c1e8-fffc-4a7e-9f22-670b8e3a4618%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/d0a4c1e8-fffc-4a7e-9f22-670b8e3a4618%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d0a4c1e8-fffc-4a7e-9f22-670b8e3a4618%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87DE4E15-0D0C-4B16-9647-FC42B7330C26%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/87DE4E15-0D0C-4B16-9647-FC42B7330C26%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marria](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@Marria](https://discuss.elastic.co/u/Marria)\ **Post date:** [February 16, 2015, 1:48pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/5 "2015-02-16T13:48:42Z")
I am really new in using ElasticSearch (I have only a week using it), and my google search didn't take me to this code. So, if they are extracted, how can I get them again (be patient with my ignorance please ☹ ) Thanks a lot for your help sir -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cab521d8-d341-45c1-b996-89ccbc3099ac%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cab521d8-d341-45c1-b996-89ccbc3099ac%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 16, 2015, 2:18pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/6 "2015-02-16T14:18:57Z")
I think this could help: [GitHub - elastic/elasticsearch-mapper-attachments: Mapper Attachments Type plugin for Elasticsearch](https://github.com/elasticsearch/elasticsearch-mapper-attachments#querying-or-accessing-metadata) [https://github.com/elasticsearch/elasticsearch-mapper-attachments#querying-or-accessing-metadata](https://github.com/elasticsearch/elasticsearch-mapper-attachments#querying-or-accessing-metadata) Replace file.content\_type with file.keywords this should work -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > Le 16 févr. 2015 à 14:48, Marria [m\_bekrar@esi.dz](mailto:m_bekrar@esi.dz) a écrit : > > I am really new in using Elasticsearch (I have only a week using it), and my google search didn't take me to this code. > > So, if they are extracted, how can I get them again > > (be patient with my ignorance please ☹ ) > > Thanks a lot for your help sir > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cab521d8-d341-45c1-b996-89ccbc3099ac%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cab521d8-d341-45c1-b996-89ccbc3099ac%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/cab521d8-d341-45c1-b996-89ccbc3099ac%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/cab521d8-d341-45c1-b996-89ccbc3099ac%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1E486949-B55F-4DBC-ABBE-E9447336893B%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/1E486949-B55F-4DBC-ABBE-E9447336893B%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marria](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@Marria](https://discuss.elastic.co/u/Marria)\ **Post date:** [February 16, 2015, 2:52pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/7 "2015-02-16T14:52:00Z")
Hi David, Reading the code, made me conclude that I didn't explain well what I need, What I mean by an automatic extraction is not to get the keywords i already had entered in my metadata but an intelligent extraction from the text. Like Alchemy that's is based on machine learning: [http://www.alchemyapi.com/products/demo/alchemylanguage/](http://www.alchemyapi.com/products/demo/alchemylanguage/) I think, it is not possible with elasticsearch because it is not the objective of this tool ☹ Thanks a lot David for your generous help 🙂 Le lundi 16 février 2015 14:00:38 UTC+1, Marria a écrit : > Hi all, > > I started using Elasticsearch to index my corpus of PDF files, I succeeded > in indexing my PDF files as attachments (base64), my search queries on the > content go right but I couldn't find how to extract automaticaly keywords > from these files in Elasticsearch. Is it possible to do that with > Elasticsearch or not? > > Could anybody help with relevent links or advices?? > > Thanks a lot. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 16, 2015, 3:19pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/8 "2015-02-16T15:19:25Z")
May be you could build something using percolator feature? I know that some users use that to classify information. Though it’s not an automatic classification and you need to provide queries. My 0.05 cents -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > Le 16 févr. 2015 à 15:52, Marria [m\_bekrar@esi.dz](mailto:m_bekrar@esi.dz) a écrit : > > Hi David, > > Reading the code, made me conclude that I didn't explain well what I need, > > What I mean by an automatic extraction is not to get the keywords i already had entered in my metadata but an intelligent extraction from the text. Like Alchemy that's is based on machine learning: > > [http://www.alchemyapi.com/products/demo/alchemylanguage/](http://www.alchemyapi.com/products/demo/alchemylanguage/) > > I think, it is not possible with elasticsearch because it is not the objective of this tool ☹ > > Thanks a lot David for your generous help 🙂 > > Le lundi 16 février 2015 14:00:38 UTC+1, Marria a écrit : > Hi all, > > I started using Elasticsearch to index my corpus of PDF files, I succeeded in indexing my PDF files as attachments (base64), my search queries on the content go right but I couldn't find how to extract automaticaly keywords from these files in Elasticsearch. Is it possible to do that with Elasticsearch or not? > > Could anybody help with relevent links or advices?? > > Thanks a lot. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2E937850-6847-417E-970E-D5A460CF6C41%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/2E937850-6847-417E-970E-D5A460CF6C41%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Charlie\_Hull](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charlie\_Hull](https://discuss.elastic.co/u/Charlie_Hull)\ **Post date:** [February 16, 2015, 3:28pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/9 "2015-02-16T15:28:16Z")
On 16/02/2015 14:52, Marria wrote: > Hi David, > > Reading the code, made me conclude that I didn't explain well what I need, > > What I mean by an automatic extraction is not to get the keywords i > already had entered in my metadata but an intelligent extraction from > the text. Like Alchemy that's is based on machine learning: > > [http://www.alchemyapi.com/products/demo/alchemylanguage/](http://www.alchemyapi.com/products/demo/alchemylanguage/) > > I think, it is not possible with elasticsearch because it is not the > objective of this tool ☹ > > Thanks a lot David for your generous help 🙂 Hi Marria, Firstly, why do you need to extract the keywords? Are you trying to extract entities (e.g. company names, people, places), tag for sentiment, or do term expansion (automatically add synonyms or related terms)? We've used Stanford NLP successfully for entity extraction and basic sentiment tagging [http://nlp.stanford.edu/](http://nlp.stanford.edu/) Python NLTK is another option [http://www.nltk.org/](http://www.nltk.org/) You're right, this isn't a core function of Elasticsearch, but rather something you would do at index time to enhance the data before you index it, or at query time to enhance a query before you use it on the index. You should also bear in mind that most of these tools only have a certain success rate, may need training and may have a significant overhead. Certainly take with a very large pinch of salt any claims of 'intelligence' especially from closed-source vendors. HTH Cheers Charlie > Le lundi 16 février 2015 14:00:38 UTC+1, Marria a écrit : > > ``` > Hi all, > > I started using ElasticSearch to index my corpus of PDF files, I > succeeded in indexing my PDF files as attachments (base64), my > search queries on the content go right but I couldn't find how to > extract automaticaly keywords from these files in ElasticSearch. Is > it possible to do that with ElasticSearch or not? > > Could anybody help with relevent links or advices?? > > Thanks a lot. > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Charlie Hull Flax - Open Source Enterprise Search tel/fax: +44 (0)8700 118334 mobile: +44 (0)7767 825828 web: [www.flax.co.uk](http://www.flax.co.uk) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/54E20C90.4030008%40flax.co.uk](https://groups.google.com/d/msgid/elasticsearch/54E20C90.4030008%40flax.co.uk). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marria](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@Marria](https://discuss.elastic.co/u/Marria)\ **Post date:** [February 16, 2015, 4:21pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/10 "2015-02-16T16:21:42Z")
Ok David, i'll search and see what it gives 🙂 thanks a lot 🙂 Le lundi 16 février 2015 16:19:49 UTC+1, David Pilato a écrit : > May be you could build something using percolator feature? > I know that some users use that to classify information. Though it’s not > an automatic classification and you need to provide queries. > > My 0.05 cents > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com) > [http://Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 16 févr. 2015 à 15:52, Marria \\> a écrit : > > Hi David, > > Reading the code, made me conclude that I didn't explain well what I need, > > What I mean by an automatic extraction is not to get the keywords i > already had entered in my metadata but an intelligent extraction from the > text. Like Alchemy that's is based on machine learning: > > [http://www.alchemyapi.com/products/demo/alchemylanguage/](http://www.alchemyapi.com/products/demo/alchemylanguage/) > > I think, it is not possible with elasticsearch because it is not the > objective of this tool ☹ > > Thanks a lot David for your generous help 🙂 > > Le lundi 16 février 2015 14:00:38 UTC+1, Marria a écrit : > > > Hi all, > > > > I started using Elasticsearch to index my corpus of PDF files, I > > succeeded in indexing my PDF files as attachments (base64), my search > > queries on the content go right but I couldn't find how to extract > > automaticaly keywords from these files in Elasticsearch. Is it possible to > > do that with Elasticsearch or not? > > > > Could anybody help with relevent links or advices?? > > > > Thanks a lot. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0eea7804-e37b-494f-8c7f-4a70a723ff4a%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a3a1b3a9-235b-4bc9-a4c7-e7bc5d10c428%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3a1b3a9-235b-4bc9-a4c7-e7bc5d10c428%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marria](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@Marria](https://discuss.elastic.co/u/Marria)\ **Post date:** [February 16, 2015, 4:27pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/11 "2015-02-16T16:27:31Z")
Hi charlie, Thanks for your help, I need to extract important keywords, themes, topics, place names from any text (the current corpus I have is a group of PDF files). I work with a research team and they want to do this on the scientific papers they exploit. For example, in a document about business plan of a mobile application developed in a certain company, I need to be able to sort as keywords "Name of the company", "mobile", "application", business plan", "technology"; etc. I think it is possible to do this with Stanford NLP, right? Thanks again 🙂 > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a5d86283-33f3-4262-9460-5528bdeffd33%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a5d86283-33f3-4262-9460-5528bdeffd33%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Charlie\_Hull](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charlie\_Hull](https://discuss.elastic.co/u/Charlie_Hull)\ **Post date:** [February 17, 2015, 9:44am UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/12 "2015-02-17T09:44:17Z")
On 16/02/2015 16:27, Marria wrote: > Hi charlie, > > Thanks for your help, > > I need to extract important keywords, themes, topics, place names from > any text (the current corpus I have is a group of PDF files). > I work with a research team and they want to do this on the scientific > papers they exploit. > For example, in a document about business plan of a mobile application > developed in a certain company, I need to be able to sort as keywords > "Name of the company", > "mobile", "application", business plan", "technology"; etc. > > I think it is possible to do this with Stanford NLP, right? Partially. Use Apache Tika or pdftotext to grab the text from the files, then try Stanford NER to get the name of the company. The more generic words won't be something you can get with NER, but if they're indexed by Elasticsearch they'll be available for searching. Theme and topic extraction is a harder problem. You might want to look at Latent Semantic Indexing which promises much in this area (but is frankly hard to get working and may have low quality). Consider your use case again - _why_ do you need themes/topics and how will it help the user? Cheers Charlie > Thanks again 🙂 > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a5d86283-33f3-4262-9460-5528bdeffd33%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a5d86283-33f3-4262-9460-5528bdeffd33%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/a5d86283-33f3-4262-9460-5528bdeffd33%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a5d86283-33f3-4262-9460-5528bdeffd33%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Charlie Hull Flax - Open Source Enterprise Search tel/fax: +44 (0)8700 118334 mobile: +44 (0)7767 825828 web: [www.flax.co.uk](http://www.flax.co.uk) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/54E30D71.4070409%40flax.co.uk](https://groups.google.com/d/msgid/elasticsearch/54E30D71.4070409%40flax.co.uk). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marria](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@Marria](https://discuss.elastic.co/u/Marria)\ **Post date:** [February 17, 2015, 10:21am UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/13 "2015-02-17T10:21:03Z")
Hi Charlie, I am really grateful to you. Well, my supervisor is not available this week to ask him why he wants this. But I think he wants to classify his scientific documents by topics. He wants all this (extraction of keywords, themes/topics , classification of documents...etc) integrated in ElasticSearch directly. So, after indexation in ElasticSearch, he wants to be able to extract them (for one document or a group of documents). I am not sure if it is feasible to implement all this on ElasticSearch, I am not even sure if the ElasticSearch is the best tool for these purposes. What do you think? Cheers 🙂 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/31459a21-97c8-48e2-b6c1-c0b15e1269e1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/31459a21-97c8-48e2-b6c1-c0b15e1269e1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Charlie\_Hull](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charlie\_Hull](https://discuss.elastic.co/u/Charlie_Hull)\ **Post date:** [February 18, 2015, 9:53am UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/14 "2015-02-18T09:53:32Z")
On 17/02/2015 10:21, Marria wrote: > Hi Charlie, > > I am really grateful to you. > > Well, my supervisor is not available this week to ask him why he wants > this. But I think he wants to classify his scientific documents by topics. > > He wants all this (extraction of keywords, themes/topics , > classification of documents...etc) integrated in Elasticsearch directly. > So, after indexation in Elasticsearch, he wants to be able to extract > them (for one document or a group of documents). I am not sure if it is > feasible to implement all this on Elasticsearch, I am not even sure if > the Elasticsearch is the best tool for these purposes. You can _index_ this extra information (metadata) in Elasticsearch, and use it to search/filter/facet to extract the documents accordingly. However _creating_ this extra information is something you would do as part of the document ingestion process, before indexing with Elasticsearch. Classification is another thing entirely - there are lots of ways to do this - manual, naive Bayes, using stored expressions....but again this is adding metadata to a document. I think you have lots of things to do here, and your first step would be to understand the definition of each and what is possible and with which tool. Cheers Charlie > What do you think? > > Cheers 🙂 > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/31459a21-97c8-48e2-b6c1-c0b15e1269e1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/31459a21-97c8-48e2-b6c1-c0b15e1269e1%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/31459a21-97c8-48e2-b6c1-c0b15e1269e1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/31459a21-97c8-48e2-b6c1-c0b15e1269e1%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Charlie Hull Flax - Open Source Enterprise Search tel/fax: +44 (0)8700 118334 mobile: +44 (0)7767 825828 web: [www.flax.co.uk](http://www.flax.co.uk) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/54E4611C.7070401%40flax.co.uk](https://groups.google.com/d/msgid/elasticsearch/54E4611C.7070401%40flax.co.uk). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marria](https://avatars.discourse-cdn.com/v4/letter/m/f1d935/32.png) [@Marria](https://discuss.elastic.co/u/Marria)\ **Post date:** [February 18, 2015, 1:57pm UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/15 "2015-02-18T13:57:30Z")
Ok charlie, I understand. Thanks a lot. Le lundi 16 février 2015 14:00:38 UTC+1, Marria a écrit : > Hi all, > > I started using Elasticsearch to index my corpus of PDF files, I succeeded > in indexing my PDF files as attachments (base64), my search queries on the > content go right but I couldn't find how to extract automaticaly keywords > from these files in Elasticsearch. Is it possible to do that with > Elasticsearch or not? > > Could anybody help with relevent links or advices?? > > Thanks a lot. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7218672d-8e7b-4f6a-bec5-ba32a33a2993%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7218672d-8e7b-4f6a-bec5-ba32a33a2993%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:31am UTC](https://discuss.elastic.co/t/automatic-keywords-extraction-in-elasticsearch/22187/16 "2017-07-06T00:31:53Z")