# Text Categorization in ES **URL:** \ **Category:** Elasticsearch\ **Created:** [February 21, 2014, 9:50am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945 "2014-02-21T09:50:11Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Prashant\_Pal](https://avatars.discourse-cdn.com/v4/letter/p/b77776/32.png) [@Prashant\_Pal](https://discuss.elastic.co/u/Prashant_Pal)\ **Post date:** [February 21, 2014, 9:50am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/1 "2014-02-21T09:50:11Z")
Hi, I am looking forward to write queries w.r.t. text categorization in Elasticsearch. So is there any API exists already if not how can I proceed with that? Any help is appreciable. ---
**Author:** ![Hannes\_Korte](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hannes_korte/32/909_2.png) [@Hannes\_Korte](https://discuss.elastic.co/u/Hannes_Korte)\ **Post date:** [February 21, 2014, 7:27pm UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/2 "2014-02-21T19:27:52Z")
On 21.02.2014 10:50, prashant.agrawal wrote: > I am looking forward to write queries w.r.t. text categorization in > Elasticsearch. > So is there any API exists already if not how can I proceed with that? Hi, you could do something like kNN: > **[k-nearest neighbors algorithm](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm)** > > In statistics, the k-nearest neighbors algorithm (k-NN) is a non-parametric supervised learning method first developed by Evelyn Fix and Joseph Hodges in 1951, and later expanded by Thomas Cover. It is used for classification and regression. In both cases, the input consists of the k closest training examples in a data set. The output depends on whether k-NN is used for classification or regression: > k-NN is a type of classification where the function is only approximated locally and all computat... Simply perform an MLT query and count the categories of the top-N docs. Additionally, you could weight the categories by score. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. I don't think there is anything directly usable via the API to classify documents. Maybe there is some neat trick using aggregations in ES 1.0. Hannes -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5307A8B8.6080801%40hkorte.com](https://groups.google.com/d/msgid/elasticsearch/5307A8B8.6080801%40hkorte.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Prashant\_Pal](https://avatars.discourse-cdn.com/v4/letter/p/b77776/32.png) [@Prashant\_Pal](https://discuss.elastic.co/u/Prashant_Pal)\ **Post date:** [February 24, 2014, 6:00am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/3 "2014-02-24T06:00:46Z")
Hi Hannes, Thanks for the info , also I came to know about lingo3G/Carrot Search. So whether that could also be a solution for that? ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 24, 2014, 8:06am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/4 "2014-02-24T08:06:29Z")
Install the carrot2 plugin and see if it fits your requirments: [http://download.carrotsearch.com/lingo3g/manual/#section.es](http://download.carrotsearch.com/lingo3g/manual/#section.es) Jörg On Mon, Feb 24, 2014 at 7:00 AM, prashant.agrawal \< [prashant.agrawal@paladion.net](mailto:prashant.agrawal@paladion.net)\> wrote: > Hi Hannes, > > Thanks for the info , also I came to know about lingo3G/Carrot Search. > So whether that could also be a solution for that? > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Text-Categorization-in-ES-tp4050194p4050349.html](http://elasticsearch-users.115913.n3.nabble.com/Text-Categorization-in-ES-tp4050194p4050349.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1393221646984-4050349.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1393221646984-4050349.post%40n3.nabble.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE8hAPZvuXgPMwbRRdM%3DxuMNpff7q5LrdpnSeMM\_A--kw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE8hAPZvuXgPMwbRRdM%3DxuMNpff7q5LrdpnSeMM_A--kw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Dawid\_Weiss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dawid_weiss/32/1643_2.png) [@Dawid\_Weiss](https://discuss.elastic.co/u/Dawid_Weiss)\ **Post date:** [February 24, 2014, 8:47am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/5 "2014-02-24T08:47:51Z")
If you want classification then Carrot2/ Lingo3G won't be of much use -- in short classification is assigning an unlabeled example to a pool of (previously known or computed) labels, Lingo3G and Carrot2 are for clustering (finding "labels" in an otherwise untagged set of documents or search results). > **[Statistical classification](https://en.wikipedia.org/wiki/Statistical_classification)** > > In statistics, classification is the problem of identifying which of a set of categories (sub-populations) an observation (or observations) belongs to. Examples are assigning a given email to the "spam" or "non-spam" class, and assigning a diagnosis to a given patient based on observed characteristics of the patient (sex, blood pressure, presence or absence of certain symptoms, etc.). > Often, the individual observations are analyzed into a set of quantifiable properties, known variously as expla... > **[Cluster analysis](https://en.wikipedia.org/wiki/Cluster_analysis)** > > Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense) to each other than to those in other groups (clusters). It is a main task of exploratory data analysis, and a common technique for statistical data analysis, used in many fields, including pattern recognition, image analysis, information retrieval, bioinformatics, data compression, computer graphics and machine learning. > Cluster ... I would agree with Hannes that the simplest way to "classify" documents with an inverted index would be to use a knn-like algorithm. Dawid On Mon, Feb 24, 2014 at 9:06 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Install the carrot2 plugin and see if it fits your requirments: > [http://download.carrotsearch.com/lingo3g/manual/#section.es](http://download.carrotsearch.com/lingo3g/manual/#section.es) > > Jörg > > On Mon, Feb 24, 2014 at 7:00 AM, prashant.agrawal > [prashant.agrawal@paladion.net](mailto:prashant.agrawal@paladion.net) wrote: > > > Hi Hannes, > > > > Thanks for the info , also I came to know about lingo3G/Carrot Search. > > So whether that could also be a solution for that? > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Text-Categorization-in-ES-tp4050194p4050349.html](http://elasticsearch-users.115913.n3.nabble.com/Text-Categorization-in-ES-tp4050194p4050349.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/1393221646984-4050349.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1393221646984-4050349.post%40n3.nabble.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE8hAPZvuXgPMwbRRdM%3DxuMNpff7q5LrdpnSeMM\_A--kw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE8hAPZvuXgPMwbRRdM%3DxuMNpff7q5LrdpnSeMM_A--kw%40mail.gmail.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAM21Rt-2L7%3D0TZY\_%3D4RaqJ2UOxd9%2B3TsdS38-GexOp5HWzu-7Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAM21Rt-2L7%3D0TZY_%3D4RaqJ2UOxd9%2B3TsdS38-GexOp5HWzu-7Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Prashant\_Pal](https://avatars.discourse-cdn.com/v4/letter/p/b77776/32.png) [@Prashant\_Pal](https://discuss.elastic.co/u/Prashant_Pal)\ **Post date:** [February 26, 2014, 7:28am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/6 "2014-02-26T07:28:55Z")
Hi All, To be specific I want a query like : Searching for Laptop will automatically give result for "Dell, Sony, HP, Lenevo, Samsung..." as well. As lingo3g is used for clustering the documents so it will store the reference for above terms as well. For that I have installed Carrot2 and Lingo3g on top of ES. So what should be my query wrt lingo3g to search the specified items. Or is there anything else I have to do to make it work. ---
**Author:** ![Dawid\_Weiss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dawid_weiss/32/1643_2.png) [@Dawid\_Weiss](https://discuss.elastic.co/u/Dawid_Weiss)\ **Post date:** [February 26, 2014, 8:04am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/7 "2014-02-26T08:04:22Z")
> Searching for Laptop will automatically give result for "Dell, Sony, HP, > Lenevo, Samsung..." as well. As lingo3g is used for clustering the documents > so it will store the reference for above terms as well. There is no way to get a clear, intuitive classification like this from an unsupervised clustering algorithm. You rely on prior knowledge (that these are companies, that they produce laptops, etc.). I would use faceting and pre-tag your documents with all the labels you may wish to display in your user interface. This will be more reliable and faster. You can then add clustering on top of that as a form of "dynamic faceting" which users may use to lookup keywords/ key phrases of groups of search results not covered in regular facets. > So what should be my query wrt lingo3g to search the specified items. The plugin contains the required documentation. Like I said though, the results will be disappointing if you expect perfect ontology from raw text. Dawid -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAM21Rt\_oFKS1X\_8juC\_m%3DHAQZabCfCfKgnJ%2Bhn30pcuykYQpCA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAM21Rt_oFKS1X_8juC_m%3DHAQZabCfCfKgnJ%2Bhn30pcuykYQpCA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Prashant\_Pal](https://avatars.discourse-cdn.com/v4/letter/p/b77776/32.png) [@Prashant\_Pal](https://discuss.elastic.co/u/Prashant_Pal)\ **Post date:** [February 26, 2014, 9:19am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/8 "2014-02-26T09:19:34Z")
So it means that all the classification has to be done prior, on the basis of user defined scenario. And automatically this feature is not supported either through carrot or Lingo3g. Like we have the feature of word-delimiter, hunspell filter etc. So what all things are there we can achieve by using lingo3g? ---
**Author:** ![Dawid\_Weiss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dawid_weiss/32/1643_2.png) [@Dawid\_Weiss](https://discuss.elastic.co/u/Dawid_Weiss)\ **Post date:** [February 26, 2014, 9:36am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/9 "2014-02-26T09:36:17Z")
> So it means that all the classification has to be done prior, on the basis of > user defined scenario. For proper faceting yes -- this information would either come with each document or would be extracted statically (when indexing each document). I'm sure OpenNLP and other text mining projects have named entity recognition that would be of help here. You may want to check out Grant's book on the subject. > **[Taming Text](https://www.manning.com/books/taming-text)** > > Taming Text is a hands-on, example-driven guide to working with unstructured text in the context of real-world applications. This book explores how to automatically organize text using approaches such as full-text search, proper name recognition,... > And automatically this feature is not supported either through carrot or > Lingo3g. Like we have the feature of word-delimiter, hunspell filter etc. Feel free to try Carrot2 (and Lingo3G) on your data. Cluster labels _are_ sort of dynamic facet labels, but they are not as "ideal" as statically indexed facets. Also, they are context-dependent (they will be created from scratch for each search result). They are essentially a different tool for a different task (to get a fast glimpse into a larger window of search results, for which static facets are not indexed). Dawid -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAM21Rt9WMiVe70HGJATdARRqyFyui2GUO8j3O%2B7NC4n\_9JS3CA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAM21Rt9WMiVe70HGJATdARRqyFyui2GUO8j3O%2B7NC4n_9JS3CA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Hannes\_Korte](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hannes_korte/32/909_2.png) [@Hannes\_Korte](https://discuss.elastic.co/u/Hannes_Korte)\ **Post date:** [February 26, 2014, 10:07am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/10 "2014-02-26T10:07:49Z")
On 26.02.2014 08:28, prashy wrote: > To be specific I want a query like : > Searching for Laptop will automatically give result for "Dell, Sony, HP, > Lenevo, Samsung..." as well. I'm not sure I got that correctly. Besides the text classification we talked about, this sentence could also mean that you want to expand your query. So instead of searching only for the term "Laptop" you want the query to be expanded automatically by adding highly correlated words like "Dell", "Sony", "HP", etc. to get a broader search result. Is it like that? > **[Query expansion](https://en.wikipedia.org/wiki/Query_expansion)** > > Query expansion (QE) is the process of reformulating a given query to improve retrieval performance in information retrieval operations, particularly in the context of query understanding. > In the context of search engines, query expansion involves evaluating a user's input (what words were typed into the search query area, and sometimes other types of data) and expanding the search query to match additional documents. Query expansion involves techniques such as: > Query expansion is a methodology... Hannes -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/530DBCF5.3030106%40hkorte.com](https://groups.google.com/d/msgid/elasticsearch/530DBCF5.3030106%40hkorte.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:47am UTC](https://discuss.elastic.co/t/text-categorization-in-es/15945/11 "2017-07-06T01:47:16Z")