# Using ElasticSearch for indexing forum **URL:** \ **Category:** Elasticsearch\ **Created:** [August 21, 2011, 7:55pm UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212 "2011-08-21T19:55:38Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![S3ncha\_2](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@S3ncha\_2](https://discuss.elastic.co/u/S3ncha_2)\ **Post date:** [August 21, 2011, 7:55pm UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212/1 "2011-08-21T19:55:38Z")
Hello, I would like to use ElasticSearch to index a forum database (containing about 8 million messages) and I'm not sure how to index/ query it properly. I'm planning to index a document that would represent a single mesage on the forum. It would also have properties like ForumId, TopicId, TopicName, Subject, Author, Date Posted. If I need to search for a list of messages it will work perfectly well, but I need to retrieve a list of Topics. Basically do a grouping on TopicId (and sort on date posted or score). Is this something I can achieve with ElasticSearch (and how?) . Could nested objects help in this case? Many thanks, S3ncha ---
**Author:** ![s3ncha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s3ncha/32/3136_2.png) [@s3ncha](https://discuss.elastic.co/u/s3ncha)\ **Post date:** [August 23, 2011, 8:32pm UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212/2 "2011-08-23T20:32:29Z")
Anyone -i s it a bad idea to use ElasticSearch in this case? I really like the product and functionality it provides, but having trouble figuring out if it's going to be efficient for in this scenario (indexing forum - searching for messages and returning unique topics)? Any help would be greatly appreciated. Thanks! -- View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/Using-ElasticSearch-for-indexing-forum-tp3273421p3279233.html](http://elasticsearch-users.115913.n3.nabble.com/Using-ElasticSearch-for-indexing-forum-tp3273421p3279233.html) Sent from the ElasticSearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![S3ncha\_2](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@S3ncha\_2](https://discuss.elastic.co/u/S3ncha_2)\ **Post date:** [August 23, 2011, 9:09pm UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212/3 "2011-08-23T21:09:55Z")
Anyone - is it a bad idea to use ElasticSearch in this case? I really like the product and functionality it provides, but having trouble figuring out if it's going to be efficient for in this scenario (indexing forum - searching for messages and returning unique topics)? Any help would be greatly appreciated. Thanks! ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [August 23, 2011, 10:40pm UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212/4 "2011-08-23T22:40:39Z")
We implement a threaded forum using elasticsearch to hold data. What's your concern? -- jim ---
**Author:** ![S3ncha\_2](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@S3ncha\_2](https://discuss.elastic.co/u/S3ncha_2)\ **Post date:** [August 24, 2011, 7:08am UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212/5 "2011-08-24T07:08:31Z")
Hello James, I would like to use elasticsearch purely for the searching purpose and continue to using another database as main db that holds forum data. Are you storing individual message or whole thread (containing all messages) as document in the elastic search? I'm planning to store messages as documents, I would like to implement 2 different search options on the forum: 1. search that would return message snippets. 2. search that would return list of threads How can I implement the 2nd option? As I understand elasticsearch doesn't support field collapsing yet, so I cant group documents by thread id when doing this search. Alex ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [August 24, 2011, 4:35pm UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212/6 "2011-08-24T16:35:11Z")
I am using ES as the data store also, but that shouldn't impact the searchability you seek. Your current data structure, probably contains a unique id for each message and a parent id. I would encourage you to also include a thread id. You would index into ES a document with a structure like this: { message\_id: , parent\_id: , // not really helpful for your use case thread\_id: , message: } If you want to search by author, sort by date, etc. you will need to include those fields. If you want to provide a snippet of the message and highlight text that corresponds to the user's search terms (ala google and most good forum searches) you will "store" the message in the index. If you just want a list of message\_id's back, you do not need to store the "message" itself. If you do a search, and the same thread id appears in multiple results, but you only want the thread id to appear one time, that is what faceting is used for. Someone else with experience using facets will have to weigh in here. ---
**Author:** ![S3ncha\_2](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@S3ncha\_2](https://discuss.elastic.co/u/S3ncha_2)\ **Post date:** [August 24, 2011, 7:47pm UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212/7 "2011-08-24T19:47:26Z")
Thanks for reply. This is pretty much how I intended to store the documents. I did look at the facets, but if I use following structure { message\_id: , thread\_id: , message: posted\_date : } I don't think I will be able to search the index and return list of thread\_id (facet on thread\_id) sorted by posted\_date. I'm I missing something? I can't find any example using facets and sorting. Also I need to return a min(message\_id) for each thread\_id. In SQL(tsql for example) it would look like select thread\_id, min(message\_id) from messages where contains(message,'keyword') order by thread\_id Can I do something similar with ES? Thanks ---
**Author:** ![S3ncha\_2](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@S3ncha\_2](https://discuss.elastic.co/u/S3ncha_2)\ **Post date:** [August 24, 2011, 7:51pm UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212/8 "2011-08-24T19:51:51Z")
sorry, just realised that I didn't include group by in my sql example. it should be : select thread\_id, min(message\_id) from messages where contains(message,'keyword') group by(thread\_id) order by thread\_id ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:56am UTC](https://discuss.elastic.co/t/using-elasticsearch-for-indexing-forum/5212/9 "2017-07-06T03:56:07Z")