# Finding the right balance between SQL and ES **URL:** \ **Category:** Elasticsearch\ **Created:** [February 9, 2011, 2:29pm UTC](https://discuss.elastic.co/t/finding-the-right-balance-between-sql-and-es/3905 "2011-02-09T14:29:42Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Jondow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jondow/32/2111_2.png) [@Jondow](https://discuss.elastic.co/u/Jondow)\ **Post date:** [February 9, 2011, 2:29pm UTC](https://discuss.elastic.co/t/finding-the-right-balance-between-sql-and-es/3905/1 "2011-02-09T14:29:42Z")
We currently have a lot of data sitting in numerous tables that describe the entity that is queried, or paged (like a catalog would be). The querying part is however very brute force and I felt it could benefit from the use of a search engine like ES. The one decision I'm trying to make however is whether to further extrapolate the existing SQL tables and 'massage' the somewhat raw format of the data into other tables that would ease the burden on SQL queries, or whether ES would be completely up to the task itself, of being used to search, and also to simply page through a list of entities based on various filtering criteria (so somewhat like a search... essentially a filter is like a 'saved search' I guess). But I'm not sure whether to introduce the intermediate SQL tables as well as to use ES, or whether to simply just use ES and the SQL tables are there as backup (if you will) of the data, and to go directly to certain data once ES has produced its results. I'm not sure if I've explained myself well, but if I have, does anybody have any insights/advice on this? Thanks in advance, Darryl Pentz ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 9, 2011, 7:27pm UTC](https://discuss.elastic.co/t/finding-the-right-balance-between-sql-and-es/3905/2 "2011-02-09T19:27:27Z")
You explained it well, but the question is quite broad. You will need to do some testing and see if you can do what you want with elasticsearch. On Wednesday, February 9, 2011 at 4:29 PM, Jondow wrote: > We currently have a lot of data sitting in numerous tables that > describe the entity that is queried, or paged (like a catalog would > be). The querying part is however very brute force and I felt it could > benefit from the use of a search engine like ES. > > The one decision I'm trying to make however is whether to further > extrapolate the existing SQL tables and 'massage' the somewhat raw > format of the data into other tables that would ease the burden on SQL > queries, or whether ES would be completely up to the task itself, of > being used to search, and also to simply page through a list of > entities based on various filtering criteria (so somewhat like a > search... essentially a filter is like a 'saved search' I guess). > > But I'm not sure whether to introduce the intermediate SQL tables as > well as to use ES, or whether to simply just use ES and the SQL tables > are there as backup (if you will) of the data, and to go directly to > certain data once ES has produced its results. > > I'm not sure if I've explained myself well, but if I have, does > anybody have any insights/advice on this? > > Thanks in advance, > Darryl Pentz ---
**Author:** ![Stephane\_Bastian](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@Stephane\_Bastian](https://discuss.elastic.co/u/Stephane_Bastian)\ **Post date:** [February 10, 2011, 9:35am UTC](https://discuss.elastic.co/t/finding-the-right-balance-between-sql-and-es/3905/3 "2011-02-10T09:35:55Z")
Hi Darryl, We had the same question a while back when indexing a bunch of data that where inside a sql db. We basically ended up using Elasticsearch (in fact it was solr at that time) for pretty much _all_ queries. We store the table/id field in the ES index so that we get the actual data from the SQL store as usual. The only difference is that we use it as some sort of key/value store where the key is the row id returned by ES. This works pretty well for us Hope this helps. Stephane Bastian [http://www.presdici.fr](http://www.presdici.fr) On Wed, 2011-02-09 at 21:27 +0200, Shay Banon wrote: > You explained it well, but the question is quite broad. You will need > to do some testing and see if you can do what you want with > elasticsearch. > > On Wednesday, February 9, 2011 at 4:29 PM, Jondow wrote: > > > We currently have a lot of data sitting in numerous tables that > > describe the entity that is queried, or paged (like a catalog would > > be). The querying part is however very brute force and I felt it > > could > > benefit from the use of a search engine like ES. > > > > The one decision I'm trying to make however is whether to further > > extrapolate the existing SQL tables and 'massage' the somewhat raw > > format of the data into other tables that would ease the burden on > > SQL > > queries, or whether ES would be completely up to the task itself, of > > being used to search, and also to simply page through a list of > > entities based on various filtering criteria (so somewhat like a > > search... essentially a filter is like a 'saved search' I guess). > > > > But I'm not sure whether to introduce the intermediate SQL tables as > > well as to use ES, or whether to simply just use ES and the SQL > > tables > > are there as backup (if you will) of the data, and to go directly to > > certain data once ES has produced its results. > > > > I'm not sure if I've explained myself well, but if I have, does > > anybody have any insights/advice on this? > > > > Thanks in advance, > > Darryl Pentz ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:12am UTC](https://discuss.elastic.co/t/finding-the-right-balance-between-sql-and-es/3905/4 "2017-07-06T04:12:16Z")