# Looking for architectural direction in porting an existing table-based PoC to ES/Hadoop production **URL:** \ **Category:** Elasticsearch\ **Created:** [September 5, 2016, 1:42pm UTC](https://discuss.elastic.co/t/looking-for-architectural-direction-in-porting-an-existing-table-based-poc-to-es-hadoop-production/59817 "2016-09-05T13:42:09Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![baden0x1](https://avatars.discourse-cdn.com/v4/letter/b/4da419/32.png) [@baden0x1](https://discuss.elastic.co/u/baden0x1)\ **Post date:** [September 5, 2016, 1:42pm UTC](https://discuss.elastic.co/t/looking-for-architectural-direction-in-porting-an-existing-table-based-poc-to-es-hadoop-production/59817/1 "2016-09-05T13:42:09Z")
We have a PoC that has been successfully running in the .Net environment with a SQL Server back-end. The system manages several websites with their own respective configuration data sets, all currently managed under one tenant. The respective configuration settings are read by a data spider that aggregates filtered RSS Data into a data mart. In the Windows/.Net world, we have built a Windows service that reads the configs, crawls sources and places the data into a SQL Server. Displaying the data is done through a website that returns free-text and pre-defined, link-based searches. Seeing the benefits of ElasticSearch I am now looking on how to build a similar system under the ES umbrella and looking for some guidance on this, with this ultimate goal of using ElasitcSearch for Hadoop on the data. As I am new to this forum, please let me know if I should distribute the following questions into multiple topics: **Configuration/User info** In my research I have found that some people will still use SQL Server (or another database) for their configuration parameters, user accounts, tenant management, etc., then use ES as the data-store. If we are using ES, I would prefer to stick to one database. However, as I am new to this, I would appreciate the advice of those with experience. **Indices** The new platform will be a multi-tenant platform, with a tenant being able to create multiple connected or disconnected data-sets. In relation to indices, would each tenant get their own index, or should each data-set be in its own index? **Crawling** Our crawling service would read configuration parameters and perform scheduled crawls, consuming RSS data and depositing it into the SQL Server database. As the initial architecture was in .Net, I have considered modifying the current service to use one of the ElasticSearch's .Net clients to populate the ES database. However in doing more research, it appears that [Nutch](https://qbox.io/blog/scraping-the-web-with-nutch-for-elasticsearch) is the standard for crawlers and getting data into ES. The issue with crawling is that your IP address can get blocked, and I am curious to know how large-scale crawlers get around this. Suggestions? **Accessing Data / Visualization** At a high-level, I was wondering if it was possible to expose Kibana to various tenants, thereby giving them access to only their own data, in both real-time and historical. Is this possible? **Security** If we implement ES with a .Net wrapper, we can maintain our entire ES cluster in a secured IP environment on our VPC with API access only from our specific web servers. I realize that [Shield](https://www.elastic.co/products/shield) offers some additional security/role-based benefits. However is this necessary from the get-go? I appreciate any assistance/discussions that follow. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:22pm UTC](https://discuss.elastic.co/t/looking-for-architectural-direction-in-porting-an-existing-table-based-poc-to-es-hadoop-production/59817/2 "2017-07-05T22:22:29Z")
Closed on July 5, 2017, 10:22pm UTC