# Indexing custom Lucene documents **URL:** \ **Category:** Elasticsearch\ **Created:** [February 14, 2011, 4:11pm UTC](https://discuss.elastic.co/t/indexing-custom-lucene-documents/3927 "2011-02-14T16:11:17Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Erik\_Fassler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erik_fassler/32/114484_2.png) [@Erik\_Fassler](https://discuss.elastic.co/u/Erik_Fassler)\ **Post date:** [February 14, 2011, 4:11pm UTC](https://discuss.elastic.co/t/indexing-custom-lucene-documents/3927/1 "2011-02-14T16:11:17Z")
Hi all! I am in the planning phase for for a search application and still have to decide which search engine to use. First I have been going towards Solr but than got pointed to ElasticSearch. And I must admit, it caught my attention. From what I have seen so far, this is a great and easy to use search engine. But I have a use case I'm not quite sure if this goes along well with ES. I am doing analysis of text documents using the UIMA Framework. That is, I have a text and quite a lot of annotations. For example, my analysis would mark Names in a text as 'person', cities, lakes etc as 'location' etc. It does also some quite more complicated things (detecting relations and whatsoever). I already use a UIMA component named Lucas (Lucene CAS indexer) to create a Lucene index from my annotated texts. Now, of course, I don't want to bother with using Lucene directly but rather through an elaborate search engine. In Solr I managed to write some additional classes which take a CAS and use Lucas to build a Lucene document. This document is then given to the normal Solr indexing-process and everything's fine. Finally, my question: Can I do something similar with ElasticSearch? In the simpliest way I already have a Lucene document object I'd just like to hand to the search engine which should do the rest for me. Especially with ElasticSearch I don't know where to start. Will this work with the schemaless approach? Will routing still work? I'd really appreciate if you could point me to a mechanism so I could add such capabilities. I think I saw a plugin mechanism for ES, could this be the way to go? Another possibility would be to convert the format UIMA gives me into something ElasticSearch understands out of the box. But I don't think you can express something like Lucene position\_increment in JSON, right? Thanks for your help! Erik ---
**Author:** ![Erik\_Fassler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erik_fassler/32/114484_2.png) [@Erik\_Fassler](https://discuss.elastic.co/u/Erik_Fassler)\ **Post date:** [February 15, 2011, 10:17pm UTC](https://discuss.elastic.co/t/indexing-custom-lucene-documents/3927/2 "2011-02-15T22:17:43Z")
I have further investigated this issue and read the guide for Elasticsearch. What I intend to do is basically to pass all information about analysis directly to the search engine. I don't really need a tokenizer within ES for instance as I do (a very specialized) tokenization before. So I already know my tokens 'token1 token2 token3' before index time. Additionally, for each tokens I know some meta data. For example the Lemma, Part of Speech or Entity class. You could represent this information in such a manner: 'token1\_lemma1\_PoS1\_entity1 token2\_lemma2\_PoS2\_entity2 token2|\_emma2\_PoS1\_entity2' I'd like to index all the meta information as Lucene tokens with position\_increment zero. The issue arises that the original text is not recoverable once written down in this manner. I am not able to determine the correct position\_offsets and highlighting won't work, even if I would store the original text into another field. It is perfectly possible that I miss something. Does anyone of you have an idea how to represent my data in a format which ES could understand without losing information? Best regards, Erik Am 14.02.2011 um 17:11 schrieb Erik Fäßler: > Hi all! > > I am in the planning phase for for a search application and still have to decide which search engine to use. First I have been going towards Solr but than got pointed to Elasticsearch. And I must admit, it caught my attention. > > From what I have seen so far, this is a great and easy to use search engine. But I have a use case I'm not quite sure if this goes along well with ES. > > I am doing analysis of text documents using the UIMA Framework. That is, I have a text and quite a lot of annotations. For example, my analysis would mark Names in a text as 'person', cities, lakes etc as 'location' etc. It does also some quite more complicated things (detecting relations and whatsoever). I already use a UIMA component named Lucas (Lucene CAS indexer) to create a Lucene index from my annotated texts. Now, of course, I don't want to bother with using Lucene directly but rather through an elaborate search engine. > > In Solr I managed to write some additional classes which take a CAS and use Lucas to build a Lucene document. This document is then given to the normal Solr indexing-process and everything's fine. > > Finally, my question: Can I do something similar with Elasticsearch? In the simpliest way I already have a Lucene document object I'd just like to hand to the search engine which should do the rest for me. Especially with Elasticsearch I don't know where to start. Will this work with the schemaless approach? Will routing still work? > I'd really appreciate if you could point me to a mechanism so I could add such capabilities. I think I saw a plugin mechanism for ES, could this be the way to go? > > Another possibility would be to convert the format UIMA gives me into something Elasticsearch understands out of the box. But I don't think you can express something like Lucene position\_increment in JSON, right? > > Thanks for your help! > > Erik ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 16, 2011, 2:10am UTC](https://discuss.elastic.co/t/indexing-custom-lucene-documents/3927/3 "2011-02-16T02:10:11Z")
Not sure I completely followed your question, but, you can create the json you want to index however you like, and then, write your own analyzer that will be applied to (some) of the fields based on a format you define and process them as tokens to be indexed. On Wednesday, February 16, 2011 at 12:17 AM, Erik Fäßler wrote: > I have further investigated this issue and read the guide for Elasticsearch. > > What I intend to do is basically to pass all information about analysis directly to the search engine. I don't really need a tokenizer within ES for instance as I do (a very specialized) tokenization before. So I already know my tokens 'token1 token2 token3' before index time. Additionally, for each tokens I know some meta data. For example the Lemma, Part of Speech or Entity class. You could represent this information in such a manner: > > 'token1\_lemma1\_PoS1\_entity1 token2\_lemma2\_PoS2\_entity2 token2|\_emma2\_PoS1\_entity2' > > I'd like to index all the meta information as Lucene tokens with position\_increment zero. > The issue arises that the original text is not recoverable once written down in this manner. I am not able to determine the correct position\_offsets and highlighting won't work, even if I would store the original text into another field. > > It is perfectly possible that I miss something. Does anyone of you have an idea how to represent my data in a format which ES could understand without losing information? > > Best regards, > > Erik > > Am 14.02.2011 um 17:11 schrieb Erik Fäßler: > > > Hi all! > > > > I am in the planning phase for for a search application and still have to decide which search engine to use. First I have been going towards Solr but than got pointed to Elasticsearch. And I must admit, it caught my attention. > > > > From what I have seen so far, this is a great and easy to use search engine. But I have a use case I'm not quite sure if this goes along well with ES. > > > > I am doing analysis of text documents using the UIMA Framework. That is, I have a text and quite a lot of annotations. For example, my analysis would mark Names in a text as 'person', cities, lakes etc as 'location' etc. It does also some quite more complicated things (detecting relations and whatsoever). I already use a UIMA component named Lucas (Lucene CAS indexer) to create a Lucene index from my annotated texts. Now, of course, I don't want to bother with using Lucene directly but rather through an elaborate search engine. > > > > In Solr I managed to write some additional classes which take a CAS and use Lucas to build a Lucene document. This document is then given to the normal Solr indexing-process and everything's fine. > > > > Finally, my question: Can I do something similar with Elasticsearch? In the simpliest way I already have a Lucene document object I'd just like to hand to the search engine which should do the rest for me. Especially with Elasticsearch I don't know where to start. Will this work with the schemaless approach? Will routing still work? > > I'd really appreciate if you could point me to a mechanism so I could add such capabilities. I think I saw a plugin mechanism for ES, could this be the way to go? > > > > Another possibility would be to convert the format UIMA gives me into something Elasticsearch understands out of the box. But I don't think you can express something like Lucene position\_increment in JSON, right? > > > > Thanks for your help! > > > > Erik ---
**Author:** ![Erik\_Fassler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erik_fassler/32/114484_2.png) [@Erik\_Fassler](https://discuss.elastic.co/u/Erik_Fassler)\ **Post date:** [February 16, 2011, 8:37am UTC](https://discuss.elastic.co/t/indexing-custom-lucene-documents/3927/4 "2011-02-16T08:37:01Z")
I guess I didn't describe my problem clear enough, apologies! I think I have one core question: Can I specify several fields in ES with the same name? Reasons why I would need that follow: I am aware of the fact I can write everything into JSON form I want and that I can use my custom analyzers. My point is that the actual analysis of my documents is done _before_ indexing. I use quite sophisticated components for all kind of natural language processing (NLP). So I don't want to use the lucene analysers for tokenization, lemmatazation, PoS-Tagging etc. My question was if someone of you sees a possibility to formulate all this information in JSON. First, if I do write the additional information (e.g. lemmas) right into the document text (for intance as token\_lemma token\_lemma token\_lemma....) I would be able to cut off the lemmas by a custom analyzer in ES, but it wouldn't make any sense to store the field as the field's text isn't my original document text but an annotated form. So, ES-specific question: Is it possible in ES to create several fields of the _same_ name? Lucene allows this; you can have one field called "text" which is not stored but indexed. I would use this field for indexing the above information. Additionally, I could have another field, also called "text" which isn't indexed but stored. Here my original text would go to. In the end, searching would do fine and highlighting would as well (although one has to take great care because of the position offsets when delivering a custom tokenization). Thanks for your help! Best regards, ``` Erik ``` Am 16.02.2011 03:10, schrieb Shay Banon: > Not sure I completely followed your question, but, you can create the > json you want to index however you like, and then, write your own > analyzer that will be applied to (some) of the fields based on a > format you define and process them as tokens to be indexed. > > On Wednesday, February 16, 2011 at 12:17 AM, Erik Fäßler wrote: > > > I have further investigated this issue and read the guide for > > Elasticsearch. > > > > What I intend to do is basically to pass all information about > > analysis directly to the search engine. I don't really need a > > tokenizer within ES for instance as I do (a very specialized) > > tokenization before. So I already know my tokens 'token1 token2 > > token3' before index time. Additionally, for each tokens I know some > > meta data. For example the Lemma, Part of Speech or Entity class. You > > could represent this information in such a manner: > > > > 'token1\_lemma1\_PoS1\_entity1 token2\_lemma2\_PoS2\_entity2 > > token2|\_emma2\_PoS1\_entity2' > > > > I'd like to index all the meta information as Lucene tokens with > > position\_increment zero. > > The issue arises that the original text is not recoverable once > > written down in this manner. I am not able to determine the correct > > position\_offsets and highlighting won't work, even if I would store > > the original text into another field. > > > > It is perfectly possible that I miss something. Does anyone of you > > have an idea how to represent my data in a format which ES could > > understand without losing information? > > > > Best regards, > > > > Erik > > > > Am 14.02.2011 um 17:11 schrieb Erik Fäßler: > > > > > Hi all! > > > > > > I am in the planning phase for for a search application and still > > > have to decide which search engine to use. First I have been going > > > towards Solr but than got pointed to Elasticsearch. And I must > > > admit, it caught my attention. > > > > > > From what I have seen so far, this is a great and easy to use search > > > engine. But I have a use case I'm not quite sure if this goes along > > > well with ES. > > > > > > I am doing analysis of text documents using the UIMA Framework. That > > > is, I have a text and quite a lot of annotations. For example, my > > > analysis would mark Names in a text as 'person', cities, lakes etc > > > as 'location' etc. It does also some quite more complicated things > > > (detecting relations and whatsoever). I already use a UIMA component > > > named Lucas (Lucene CAS indexer) to create a Lucene index from my > > > annotated texts. Now, of course, I don't want to bother with using > > > Lucene directly but rather through an elaborate search engine. > > > > > > In Solr I managed to write some additional classes which take a CAS > > > and use Lucas to build a Lucene document. This document is then > > > given to the normal Solr indexing-process and everything's fine. > > > > > > Finally, my question: Can I do something similar with Elasticsearch? > > > In the simpliest way I already have a Lucene document object I'd > > > just like to hand to the search engine which should do the rest for > > > me. Especially with Elasticsearch I don't know where to start. Will > > > this work with the schemaless approach? Will routing still work? > > > I'd really appreciate if you could point me to a mechanism so I > > > could add such capabilities. I think I saw a plugin mechanism for > > > ES, could this be the way to go? > > > > > > Another possibility would be to convert the format UIMA gives me > > > into something Elasticsearch understands out of the box. But I don't > > > think you can express something like Lucene position\_increment in > > > JSON, right? > > > > > > Thanks for your help! > > > > > > Erik ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 17, 2011, 1:05am UTC](https://discuss.elastic.co/t/indexing-custom-lucene-documents/3927/5 "2011-02-17T01:05:21Z")
No, you can't have same name fields with different mapping options, but you can do it with different names, no? On Wednesday, February 16, 2011 at 10:37 AM, Erik Fäßler wrote: > I guess I didn't describe my problem clear enough, apologies! > > I think I have one core question: Can I specify several fields in ES > with the same name? Reasons why I would need that follow: > > I am aware of the fact I can write everything into JSON form I want and > that I can use my custom analyzers. > > My point is that the actual analysis of my documents is done _before_ > indexing. I use quite sophisticated components for all kind of natural > language processing (NLP). So I don't want to use the lucene analysers > for tokenization, lemmatazation, PoS-Tagging etc. > > My question was if someone of you sees a possibility to formulate all > this information in JSON. First, if I do write the additional > information (e.g. lemmas) right into the document text (for intance as > token\_lemma token\_lemma token\_lemma....) I would be able to cut off the > lemmas by a custom analyzer in ES, but it wouldn't make any sense to > store the field as the field's text isn't my original document text but > an annotated form. > > So, ES-specific question: Is it possible in ES to create several fields > of the _same_ name? Lucene allows this; you can have one field called > "text" which is not stored but indexed. I would use this field for > indexing the above information. Additionally, I could have another > field, also called "text" which isn't indexed but stored. Here my > original text would go to. In the end, searching would do fine and > highlighting would as well (although one has to take great care because > of the position offsets when delivering a custom tokenization). > > Thanks for your help! > > Best regards, > > Erik > > Am 16.02.2011 03:10, schrieb Shay Banon: > > > Not sure I completely followed your question, but, you can create the > > json you want to index however you like, and then, write your own > > analyzer that will be applied to (some) of the fields based on a > > format you define and process them as tokens to be indexed. > > > > On Wednesday, February 16, 2011 at 12:17 AM, Erik Fäßler wrote: > > > > > I have further investigated this issue and read the guide for > > > Elasticsearch. > > > > > > What I intend to do is basically to pass all information about > > > analysis directly to the search engine. I don't really need a > > > tokenizer within ES for instance as I do (a very specialized) > > > tokenization before. So I already know my tokens 'token1 token2 > > > token3' before index time. Additionally, for each tokens I know some > > > meta data. For example the Lemma, Part of Speech or Entity class. You > > > could represent this information in such a manner: > > > > > > 'token1\_lemma1\_PoS1\_entity1 token2\_lemma2\_PoS2\_entity2 > > > token2|\_emma2\_PoS1\_entity2' > > > > > > I'd like to index all the meta information as Lucene tokens with > > > position\_increment zero. > > > The issue arises that the original text is not recoverable once > > > written down in this manner. I am not able to determine the correct > > > position\_offsets and highlighting won't work, even if I would store > > > the original text into another field. > > > > > > It is perfectly possible that I miss something. Does anyone of you > > > have an idea how to represent my data in a format which ES could > > > understand without losing information? > > > > > > Best regards, > > > > > > Erik > > > > > > Am 14.02.2011 um 17:11 schrieb Erik Fäßler: > > > > > > > Hi all! > > > > > > > > I am in the planning phase for for a search application and still > > > > have to decide which search engine to use. First I have been going > > > > towards Solr but than got pointed to Elasticsearch. And I must > > > > admit, it caught my attention. > > > > > > > > From what I have seen so far, this is a great and easy to use search > > > > engine. But I have a use case I'm not quite sure if this goes along > > > > well with ES. > > > > > > > > I am doing analysis of text documents using the UIMA Framework. That > > > > is, I have a text and quite a lot of annotations. For example, my > > > > analysis would mark Names in a text as 'person', cities, lakes etc > > > > as 'location' etc. It does also some quite more complicated things > > > > (detecting relations and whatsoever). I already use a UIMA component > > > > named Lucas (Lucene CAS indexer) to create a Lucene index from my > > > > annotated texts. Now, of course, I don't want to bother with using > > > > Lucene directly but rather through an elaborate search engine. > > > > > > > > In Solr I managed to write some additional classes which take a CAS > > > > and use Lucas to build a Lucene document. This document is then > > > > given to the normal Solr indexing-process and everything's fine. > > > > > > > > Finally, my question: Can I do something similar with Elasticsearch? > > > > In the simpliest way I already have a Lucene document object I'd > > > > just like to hand to the search engine which should do the rest for > > > > me. Especially with Elasticsearch I don't know where to start. Will > > > > this work with the schemaless approach? Will routing still work? > > > > I'd really appreciate if you could point me to a mechanism so I > > > > could add such capabilities. I think I saw a plugin mechanism for > > > > ES, could this be the way to go? > > > > > > > > Another possibility would be to convert the format UIMA gives me > > > > into something Elasticsearch understands out of the box. But I don't > > > > think you can express something like Lucene position\_increment in > > > > JSON, right? > > > > > > > > Thanks for your help! > > > > > > > > Erik ---
**Author:** ![Erik\_Fassler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erik_fassler/32/114484_2.png) [@Erik\_Fassler](https://discuss.elastic.co/u/Erik_Fassler)\ **Post date:** [February 17, 2011, 9:03am UTC](https://discuss.elastic.co/t/indexing-custom-lucene-documents/3927/6 "2011-02-17T09:03:22Z")
Would highlighting still work when I have an indexed field "text\_terms" with all my annotation information and a field "text" which only holds the original text? I would search on "text\_terms" and would like to retrieve a highlighted portion of the "text" value. Does this work? Am 17.02.2011 02:05, schrieb Shay Banon: > No, you can't have same name fields with different mapping options, > but you can do it with different names, no? > > On Wednesday, February 16, 2011 at 10:37 AM, Erik Fäßler wrote: > > > I guess I didn't describe my problem clear enough, apologies! > > > > I think I have one core question: Can I specify several fields in ES > > with the same name? Reasons why I would need that follow: > > > > I am aware of the fact I can write everything into JSON form I want and > > that I can use my custom analyzers. > > > > My point is that the actual analysis of my documents is done _before_ > > indexing. I use quite sophisticated components for all kind of natural > > language processing (NLP). So I don't want to use the lucene analysers > > for tokenization, lemmatazation, PoS-Tagging etc. > > > > My question was if someone of you sees a possibility to formulate all > > this information in JSON. First, if I do write the additional > > information (e.g. lemmas) right into the document text (for intance as > > token\_lemma token\_lemma token\_lemma....) I would be able to cut off the > > lemmas by a custom analyzer in ES, but it wouldn't make any sense to > > store the field as the field's text isn't my original document text but > > an annotated form. > > > > So, ES-specific question: Is it possible in ES to create several fields > > of the _same_ name? Lucene allows this; you can have one field called > > "text" which is not stored but indexed. I would use this field for > > indexing the above information. Additionally, I could have another > > field, also called "text" which isn't indexed but stored. Here my > > original text would go to. In the end, searching would do fine and > > highlighting would as well (although one has to take great care because > > of the position offsets when delivering a custom tokenization). > > > > Thanks for your help! > > > > Best regards, > > > > Erik > > > > Am 16.02.2011 03:10, schrieb Shay Banon: > > > > > Not sure I completely followed your question, but, you can create the > > > json you want to index however you like, and then, write your own > > > analyzer that will be applied to (some) of the fields based on a > > > [ZEIT ONLINE | Nachrichten, News, Hintergründe und Debatten](http://www.zeit.de/index) > > > format you define and process them as tokens to be indexed. > > > > > > On Wednesday, February 16, 2011 at 12:17 AM, Erik Fäßler wrote: > > > > > > > I have further investigated this issue and read the guide for > > > > Elasticsearch. > > > > > > > > What I intend to do is basically to pass all information about > > > > analysis directly to the search engine. I don't really need a > > > > tokenizer within ES for instance as I do (a very specialized) > > > > tokenization before. So I already know my tokens 'token1 token2 > > > > token3' before index time. Additionally, for each tokens I know some > > > > meta data. For example the Lemma, Part of Speech or Entity class. You > > > > could represent this information in such a manner: > > > > > > > > 'token1\_lemma1\_PoS1\_entity1 token2\_lemma2\_PoS2\_entity2 > > > > token2|\_emma2\_PoS1\_entity2' > > > > > > > > I'd like to index all the meta information as Lucene tokens with > > > > position\_increment zero. > > > > The issue arises that the original text is not recoverable once > > > > written down in this manner. I am not able to determine the correct > > > > position\_offsets and highlighting won't work, even if I would store > > > > the original text into another field. > > > > > > > > It is perfectly possible that I miss something. Does anyone of you > > > > have an idea how to represent my data in a format which ES could > > > > understand without losing information? > > > > > > > > Best regards, > > > > > > > > Erik > > > > > > > > Am 14.02.2011 um 17:11 schrieb Erik Fäßler: > > > > > > > > > Hi all! > > > > > > > > > > I am in the planning phase for for a search application and still > > > > > have to decide which search engine to use. First I have been going > > > > > towards Solr but than got pointed to Elasticsearch. And I must > > > > > admit, it caught my attention. > > > > > > > > > > From what I have seen so far, this is a great and easy to use search > > > > > engine. But I have a use case I'm not quite sure if this goes along > > > > > well with ES. > > > > > > > > > > I am doing analysis of text documents using the UIMA Framework. That > > > > > is, I have a text and quite a lot of annotations. For example, my > > > > > analysis would mark Names in a text as 'person', cities, lakes etc > > > > > as 'location' etc. It does also some quite more complicated things > > > > > (detecting relations and whatsoever). I already use a UIMA component > > > > > named Lucas (Lucene CAS indexer) to create a Lucene index from my > > > > > annotated texts. Now, of course, I don't want to bother with using > > > > > Lucene directly but rather through an elaborate search engine. > > > > > > > > > > In Solr I managed to write some additional classes which take a CAS > > > > > and use Lucas to build a Lucene document. This document is then > > > > > given to the normal Solr indexing-process and everything's fine. > > > > > > > > > > Finally, my question: Can I do something similar with Elasticsearch? > > > > > In the simpliest way I already have a Lucene document object I'd > > > > > just like to hand to the search engine which should do the rest for > > > > > me. Especially with Elasticsearch I don't know where to start. Will > > > > > this work with the schemaless approach? Will routing still work? > > > > > I'd really appreciate if you could point me to a mechanism so I > > > > > could add such capabilities. I think I saw a plugin mechanism for > > > > > ES, could this be the way to go? > > > > > > > > > > Another possibility would be to convert the format UIMA gives me > > > > > into something Elasticsearch understands out of the box. But I don't > > > > > think you can express something like Lucene position\_increment in > > > > > JSON, right? > > > > > > > > > > Thanks for your help! > > > > > > > > > > Erik ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:11am UTC](https://discuss.elastic.co/t/indexing-custom-lucene-documents/3927/7 "2017-07-06T04:11:55Z")