# Architecture for Elastic Search platform **URL:** \ **Category:** Elasticsearch\ **Created:** [April 3, 2012, 1:42pm UTC](https://discuss.elastic.co/t/architecture-for-elastic-search-platform/7207 "2012-04-03T13:42:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Kenny\_Meyer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenny_meyer/32/2920_2.png) [@Kenny\_Meyer](https://discuss.elastic.co/u/Kenny_Meyer)\ **Post date:** [April 3, 2012, 1:42pm UTC](https://discuss.elastic.co/t/architecture-for-elastic-search-platform/7207/1 "2012-04-03T13:42:53Z")
I am working on populating a search index powered by Elastic Search from a database hosted on the server of Team A; but I am uncertain about if Team A can push data from their servers to our search index as JSON, periodically. To give you more context: The database itself is running on top of PostgreSQL, and the it is owned by Team A. The search index is hosted on our servers, and I have full ownership over it. According to my research with Google Search, I have found "Rivers" ([http://www.elasticsearch.org/guide/reference/river/](http://www.elasticsearch.org/guide/reference/river/)), but there does not seem to be anyone supporting PostgreSQL databases. Why? Looking forward to your insights on the topic and eventually some feedback. πŸ™‚ Kenny %-) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 3, 2012, 2:34pm UTC](https://discuss.elastic.co/t/architecture-for-elastic-search-platform/7207/2 "2012-04-03T14:34:48Z")
You will need to write code that get the data from the database and index it into elasticsearch. You might want to move it to a river later on, but its not relevant now. On Tue, Apr 3, 2012 at 4:42 PM, Kenny Meyer [knny.myer@gmail.com](mailto:knny.myer@gmail.com) wrote: > I am working on populating a search index powered by Elastic Search from a > database hosted on the server of Team A; but I am uncertain about if Team A > can push data from their servers to our search index as JSON, periodically. > > To give you more context: > > The database itself is running on top of PostgreSQL, and the it is owned > by Team A. The search index is hosted on our servers, and I have full > ownership over it. > > According to my research with Google Search, I have found "Rivers" ( > [Elasticsearch Platform β€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/river/)), but there does not > seem to be anyone supporting PostgreSQL databases. Why? > > Looking forward to your insights on the topic and eventually some > feedback. πŸ™‚ > > Kenny %-) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 3, 2012, 2:59pm UTC](https://discuss.elastic.co/t/architecture-for-elastic-search-platform/7207/3 "2012-04-03T14:59:20Z")
You could also look at the Scrutineer project : [https://github.com/Aconex/scrutineer](https://github.com/Aconex/scrutineer) [https://github.com/Aconex/scrutineer](https://github.com/Aconex/scrutineer) It could help you for that need. David. Le 3 avril 2012 Γ  16:34, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) a Γ©crit : > You will need to write code that get the data from the database and index > it into elasticsearch. You might want to move it to a river later on, but > its not relevant now. > > On Tue, Apr 3, 2012 at 4:42 PM, Kenny Meyer \< [knny.myer@gmail.com](mailto:knny.myer@gmail.com) > [mailto:knny.myer@gmail.com](mailto:knny.myer@gmail.com) \> wrote: > > > index powered by Elastic Search from a database hosted on the server of > > Team A; but I am uncertain about if Team A can push data from their servers > > to our search index as JSON, periodically. > > > > To give you more context: > > > > The database itself is running on top of PostgreSQL, and the it is owned by > > Team A. The search index is hosted on our servers, and I have full > > ownership over it. > > > > According to my research with Google Search, I have found "Rivers" ( > > [Elasticsearch Platform β€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/river/) > > [http://www.elasticsearch.org/guide/reference/river/](http://www.elasticsearch.org/guide/reference/river/) ), but there does not > > seem to be anyone supporting PostgreSQL databases. Why? > > > > Looking forward to your insights on the topic and eventually some feedback. > > πŸ™‚ > > > > Kenny %-) -- David Pilato [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : @dadoonet ---
**Author:** ![Kenny\_Meyer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenny_meyer/32/2920_2.png) [@Kenny\_Meyer](https://discuss.elastic.co/u/Kenny_Meyer)\ **Post date:** [April 4, 2012, 11:55am UTC](https://discuss.elastic.co/t/architecture-for-elastic-search-platform/7207/4 "2012-04-04T11:55:36Z")
I tried to use it, but I can't get the connection with the database running. On Tuesday, April 3, 2012 11:59:20 AM UTC-3, David Pilato wrote: > You could also look at the Scrutineer project : > [https://github.com/Aconex/scrutineer](https://github.com/Aconex/scrutineer) > > It could help you for that need. > > David. > > Le 3 avril 2012 Γ  16:34, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) a Γ©crit : > > You will need to write code that get the data from the database and > index it into elasticsearch. You might want to move it to a river later on, > but its not relevant now. > > On Tue, Apr 3, 2012 at 4:42 PM, Kenny Meyer \< [knny.myer@gmail.com](mailto:knny.myer@gmail.com) \> wrote: > > I am working on populating a search index powered by Elastic Search from a > database hosted on the server of Team A; but I am uncertain about if Team A > can push data from their servers to our search index as JSON, periodically. > > To give you more context: > > The database itself is running on top of PostgreSQL, and the it is owned > by Team A. The search index is hosted on our servers, and I have full > ownership over it. > > According to my research with Google Search, I have found "Rivers" ( > [Elasticsearch Platform β€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/river/) ), but there does not > seem to be anyone supporting PostgreSQL databases. Why? > > Looking forward to your insights on the topic and eventually some > feedback. πŸ™‚ > > Kenny %-) > > -- > David Pilato > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > Twitter : @dadoonet ---
**Author:** ![Kenny\_Meyer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenny_meyer/32/2920_2.png) [@Kenny\_Meyer](https://discuss.elastic.co/u/Kenny_Meyer)\ **Post date:** [April 4, 2012, 12:03pm UTC](https://discuss.elastic.co/t/architecture-for-elastic-search-platform/7207/5 "2012-04-04T12:03:58Z")
I wrote a small and super-simple PHP indexer, and I am open to other, existing approaches. > I find it strange that there's not a lot of code out there for indexing data from a RDBMS. May it be that Elastic Search wasn't made for our needs, or how shall I explain that? On Tuesday, April 3, 2012 11:34:48 AM UTC-3, kimchy wrote: > You will need to write code that get the data from the database and index > it into elasticsearch. You might want to move it to a river later on, but > its not relevant now. > > On Tue, Apr 3, 2012 at 4:42 PM, Kenny Meyer [knny.myer@gmail.com](mailto:knny.myer@gmail.com) wrote: > > > I am working on populating a search index powered by Elastic Search from > > a database hosted on the server of Team A; but I am uncertain about if Team > > A can push data from their servers to our search index as JSON, > > periodically. > > > > To give you more context: > > > > The database itself is running on top of PostgreSQL, and the it is owned > > by Team A. The search index is hosted on our servers, and I have full > > ownership over it. > > > > According to my research with Google Search, I have found "Rivers" ( > > [Elasticsearch Platform β€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/river/)), but there does not > > seem to be anyone supporting PostgreSQL databases. Why? > > > > Looking forward to your insights on the topic and eventually some > > feedback. πŸ™‚ > > > > Kenny %-) ---
**Author:** ![Greg\_Brown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greg_brown/32/1373_2.png) [@Greg\_Brown](https://discuss.elastic.co/u/Greg_Brown)\ **Post date:** [April 4, 2012, 4:29pm UTC](https://discuss.elastic.co/t/architecture-for-elastic-search-platform/7207/6 "2012-04-04T16:29:20Z")
Your PHP code looks like it would work fine as long as you continue to do a 1 to 1 mapping of RDBMS fields to ES fields. However, I'd highly recommend looking at the Elastica client library if you are building any significant amount of PHP indexing/searching: [GitHub - ruflin/Elastica: Elastica is a PHP client for elasticsearch](https://github.com/ruflin/Elastica) Great library. Makes the code much more readable/maintainable. We pull all of our data from RDBMS, but generally do a fair bit of massaging of the data before putting it into ES. I would expect that is true of many users and that is the reason why a generic way of pulling from an RDBMS doesn't exist. In particular we end up adding new fields to search on, strip HTML in some cases, and take constants and turn them into names. The way a user wants to search is rarely the way that you store data in an RDBMS so a 1 to 1 mapping won't always work. -Greg On Apr 4, 6:03 am, Kenny Meyer [knny.m...@gmail.com](mailto:knny.m...@gmail.com) wrote: > I wrote a small and super-simple PHP indexer, and I am open to other, > existing approaches. > > [Elastic Search indexer Β· GitHub](https://gist.github.com/2300615) > > I find it strange that there's not a lot of code out there for indexing > data from a RDBMS. May it be that Elastic Search wasn't made for our needs, > or how shall I explain that? > > On Tuesday, April 3, 2012 11:34:48 AM UTC-3, kimchy wrote: > > > You will need to write code that get the data from the database and index > > it into elasticsearch. You might want to move it to a river later on, but > > its not relevant now. > > > On Tue, Apr 3, 2012 at 4:42 PM, Kenny Meyer [knny.m...@gmail.com](mailto:knny.m...@gmail.com) wrote: > > > > I am working on populating a search index powered by Elastic Search from > > > a database hosted on the server of Team A; but I am uncertain about if Team > > > A can push data from their servers to our search index as JSON, > > > periodically. > > > > To give you more context: > > > > The database itself is running on top of PostgreSQL, and the it is owned > > > by Team A. The search index is hosted on our servers, and I have full > > > ownership over it. > > > > According to my research with Google Search, I have found "Rivers" ( > > > [Elasticsearch Platform β€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/river/)), but there does not > > > seem to be anyone supporting PostgreSQL databases. Why? > > > > Looking forward to your insights on the topic and eventually some > > > feedback. πŸ™‚ > > > > Kenny %-) ---
**Author:** ![Kenny\_Meyer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenny_meyer/32/2920_2.png) [@Kenny\_Meyer](https://discuss.elastic.co/u/Kenny_Meyer)\ **Post date:** [April 13, 2012, 1:15pm UTC](https://discuss.elastic.co/t/architecture-for-elastic-search-platform/7207/7 "2012-04-13T13:15:44Z")
Greg, thanks for the advice. I have totally missed your reply somehow. Do you normalize the data from the RDBMS inside of Elastica? On Tuesday, April 3, 2012 10:42:53 AM UTC-3, Kenny Meyer wrote: > I am working on populating a search index powered by Elastic Search from a > database hosted on the server of Team A; but I am uncertain about if Team A > can push data from their servers to our search index as JSON, periodically. > > To give you more context: > > The database itself is running on top of PostgreSQL, and the it is owned > by Team A. The search index is hosted on our servers, and I have full > ownership over it. > > According to my research with Google Search, I have found "Rivers" ( > [Elasticsearch Platform β€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/river/)), but there does not > seem to be anyone supporting PostgreSQL databases. Why? > > Looking forward to your insights on the topic and eventually some > feedback. πŸ™‚ > > Kenny %-) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:32am UTC](https://discuss.elastic.co/t/architecture-for-elastic-search-platform/7207/8 "2017-07-06T03:32:40Z")