# Indexing web pages **URL:** \ **Category:** Elasticsearch\ **Created:** [June 22, 2013, 6:12am UTC](https://discuss.elastic.co/t/indexing-web-pages/12518 "2013-06-22T06:12:28Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Zoran\_Jeremic](https://avatars.discourse-cdn.com/v4/letter/z/7ba0ec/32.png) [@Zoran\_Jeremic](https://discuss.elastic.co/u/Zoran_Jeremic)\ **Post date:** [June 22, 2013, 6:12am UTC](https://discuss.elastic.co/t/indexing-web-pages/12518/1 "2013-06-22T06:12:28Z")
Hi, I'm trying to provide indexing of Web pages so it can be later searched based on it's content. The problem is that I don't know the appropriate approach to deal with content. I tried to use attachment plugin or to store the content as document field. The problem is that either this field is not stored at all or it can't be searched. If I encode InputStream as base64 input stream, the content is stored to ES, but can't be searched. If I convert InputStream to bytes array nothing is stored to the content field. The only way I managed to make this is to store web page as the local file and then to index it as attachment. Could you please suggest me appropriate approach or give me an example how should I perform this? Thanks, Zoran -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 22, 2013, 6:22am UTC](https://discuss.elastic.co/t/indexing-web-pages/12518/2 "2013-06-22T06:22:51Z")
Could you Gist what you did using the attachment plugin? Because it should work. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 22 juin 2013 à 08:12, Zoran Jeremic [zoran.jeremic@gmail.com](mailto:zoran.jeremic@gmail.com) a écrit : Hi, I'm trying to provide indexing of Web pages so it can be later searched based on it's content. The problem is that I don't know the appropriate approach to deal with content. I tried to use attachment plugin or to store the content as document field. The problem is that either this field is not stored at all or it can't be searched. If I encode InputStream as base64 input stream, the content is stored to ES, but can't be searched. If I convert InputStream to bytes array nothing is stored to the content field. The only way I managed to make this is to store web page as the local file and then to index it as attachment. Could you please suggest me appropriate approach or give me an example how should I perform this? ## Thanks, Zoran You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ben\_Hundley](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben_hundley/32/1853_2.png) [@Ben\_Hundley](https://discuss.elastic.co/u/Ben_Hundley)\ **Post date:** [June 22, 2013, 3:41pm UTC](https://discuss.elastic.co/t/indexing-web-pages/12518/3 "2013-06-22T15:41:27Z")
Is there any reason the webpage field couldn't be stored as a String? That's what I've done personally in the past to store HTML content. ---
**Author:** ![Ben\_Hundley](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben_hundley/32/1853_2.png) [@Ben\_Hundley](https://discuss.elastic.co/u/Ben_Hundley)\ **Post date:** [June 22, 2013, 3:45pm UTC](https://discuss.elastic.co/t/indexing-web-pages/12518/4 "2013-06-22T15:45:49Z")
Ahh, I see now you're wanting to actually tokenize parts of the HTML document to make it searchable. If I were you, I'd just strip out the HTML (taking just the text content of the page) and use the standard tokenizer. That should allow the searching of pages based on their textual content, though I can't attest to performance considering webpages can be quite large. You might be able to provide a more strict pattern-based tokenizer to cut down on the size of indexed content. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 22, 2013, 4:14pm UTC](https://discuss.elastic.co/t/indexing-web-pages/12518/5 "2013-06-22T16:14:23Z")
What does mapper attachment and Tika under the scene is to get metadata from HTML and index them as fields. You can do that yourself also. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 22 juin 2013 à 17:45, Ben Hundley [ben@qbox.io](mailto:ben@qbox.io) a écrit : Ahh, I see now you're wanting to actually tokenize parts of the HTML document to make it searchable. If I were you, I'd just strip out the HTML (taking just the text content of the page) and use the standard tokenizer. That should allow the searching of pages based on their textual content, though I can't attest to performance considering webpages can be quite large. You might be able to provide a more strict pattern-based tokenizer to cut down on the size of indexed content. * * * -- View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/Indexing-web-pages-tp4036976p4036985.html](http://elasticsearch-users.115913.n3.nabble.com/Indexing-web-pages-tp4036976p4036985.html) Sent from the ElasticSearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Zoran\_Jeremic](https://avatars.discourse-cdn.com/v4/letter/z/7ba0ec/32.png) [@Zoran\_Jeremic](https://discuss.elastic.co/u/Zoran_Jeremic)\ **Post date:** [June 23, 2013, 7:08am UTC](https://discuss.elastic.co/t/indexing-web-pages/12518/6 "2013-06-23T07:08:11Z")
Hi guys, > > Could you Gist what you did using the attachment plugin? > > Because it should work. I committed it to the bitbucket [Bitbucket](https://bitbucket.org/zjeremic/elastic-search-test-application). It stores the documents, but I can't search over it and find the terms that exists in the web page content. I tried both with Elasticsearch Head and Java client and content from web page could not be found. However, storing the pdf or other document, even html page saved on the pc works fine and I can search over the content and find the terms. > > Ahh, I see now you're wanting to actually tokenize parts of the HTML > > document > > to make it searchable. I'm actually trying to implement a recommendation service. The scenario I have is as follows: Application is learning system where students could upload documents or links to the interesting web pages. These documents and web pages could be later recommended to other students if they are relevant for what student is currently working on. I have already implemented this before with Lucene and Tika and what I actually have done is to extract a Vector of Terms related to the student's current context he is working on and then compare this vector with all documents stored in Lucene repository. I was using MoreLikeThis to find similar documents to the provided vector of terms. All stored documents (web pages) were describe by the vector of Most Interested Terms I have extracted previously during the indexing of the Web Page. I didn't like this approach very much as I had to re-index all the documents after a certain period of time due to the IDF change, as it affected a list of Interested Terms for a specific document as the repository was populated with new documents. I thought that it could be possible to tokenize the content of the Web page and use it instead of the interested terms if the performance will be good still. I'm not sure how the ES works at the backend and how does the content of the web page is stored if I provide InputStream as the content. Will this plugin do the rest of the job and clean it from the html code or I should take care about this? Is it better to use the old approach and extract Interested Terms from the content or I can store the whole web page content? I know Lucene's MoreLikeThis has a function getInterestingTerms to extract it, but I couldn't find anything similar in ES. Is it implemented? I would appreciate if you give me your suggestions about this? Best, Zoran On Friday, 21 June 2013 23:12:28 UTC-7, Zoran Jeremic wrote: > Hi, > > I'm trying to provide indexing of Web pages so it can be later searched > based on it's content. The problem is that I don't know the appropriate > approach to deal with content. I tried to use attachment plugin or to store > the content as document field. The problem is that either this field is not > stored at all or it can't be searched. If I encode InputStream as base64 > input stream, the content is stored to ES, but can't be searched. If I > convert InputStream to bytes array nothing is stored to the content field. > The only way I managed to make this is to store web page as the local file > and then to index it as attachment. > > Could you please suggest me appropriate approach or give me an example how > should I perform this? > > Thanks, > Zoran -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 23, 2013, 1:15pm UTC](https://discuss.elastic.co/t/indexing-web-pages/12518/7 "2013-06-23T13:15:01Z")
Zorane, Sounds like you should first take a look at Nutch or Droids from Apache. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Saturday, June 22, 2013 2:12:28 AM UTC-4, Zoran Jeremic wrote: > Hi, > > I'm trying to provide indexing of Web pages so it can be later searched > based on it's content. The problem is that I don't know the appropriate > approach to deal with content. I tried to use attachment plugin or to store > the content as document field. The problem is that either this field is not > stored at all or it can't be searched. If I encode InputStream as base64 > input stream, the content is stored to ES, but can't be searched. If I > convert InputStream to bytes array nothing is stored to the content field. > The only way I managed to make this is to store web page as the local file > and then to index it as attachment. > > Could you please suggest me appropriate approach or give me an example how > should I perform this? > > Thanks, > Zoran -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:29am UTC](https://discuss.elastic.co/t/indexing-web-pages/12518/8 "2017-07-06T02:29:53Z")