# Search across multiple ES data sources **URL:** \ **Category:** Elasticsearch\ **Created:** [March 12, 2019, 1:32pm UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959 "2019-03-12T13:32:46Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 2, 2019, 10:34am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/21 "2019-04-02T10:34:51Z")
> [@vasekz](#): > > as the URL is limited by ~2K symbols TIL there _is_ a limit. It defaults to 4kB but can be adjusted by changing `http.max_initial_line_length`. If you are seeing a limit of ~2k then this is being imposed by something outside of Elasticsearch. Also I agree with what @Christian_Dahlqvist said. Your proposed architecture is very strange. ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 10:41am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/22 "2019-04-02T10:41:40Z")
The requirements are more-or-less like this: There are hundreds, possibly tens of thousands, depending on the deployment, devices that report data. The devices are deployed on premise. I need to perform queries and analytics on this data at some central place and also locally in the standalone device context. Each device produces up to 100K key-value pairs, where value is itself a map of ~200 key-value pairs, where each such value is a string of size 100 bytes in average. ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 10:47am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/23 "2019-04-02T10:47:11Z")
> [@DavidTurner](#): > > our proposed architecture is very strange. Please clarify why it's strange. Regarding 2K limit, it's a browser's limit and I cannot ask the customer to change it. More precisely, I would prefer to to ask until it's absolutely necessary. ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 11:01am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/24 "2019-04-02T11:01:10Z")
Well, I understand that URL is eventually not limited. _As Elasticsearch can be quite resource intensive, creating a few geographically distributed and highly available clusters that can be queries across CCS is the normal deployment pattern_ Does this mean that a large number of nodes for CCS is not acceptable? How about my algorithm above that works essentially the same (if it is 🙂 ) as CCS but without it? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 11:06am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/25 "2019-04-02T11:06:09Z")
Elasticsearch is designed to be a server component and requires a fair bit of resources. Are the devices able to support running Elasticsearch locally? ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 11:06am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/26 "2019-04-02T11:06:46Z")
yes. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 11:11am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/27 "2019-04-02T11:11:59Z")
Sounds like a very strange way to deploy Elasticsearch and I have seen anything that comes even remotely close. I would therefore recommend against deploying it this way as you are likely to face issues and potential limitations that no one has come across before. ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 11:13am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/28 "2019-04-02T11:13:29Z")
I do not understand... Is the number of nodes too large? What is strange? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 11:15am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/29 "2019-04-02T11:15:39Z")
Trying to connect a very large number of single node clusters is not a common deployment pattern. ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 11:17am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/30 "2019-04-02T11:17:39Z")
What would you suggest then? Nodes are not necessary connected to each other via the network. They produce the data independently and I have to present an aggregated view. The connection between the nodes and the central point is promised. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 11:21am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/31 "2019-04-02T11:21:57Z")
Without more details about the devices and their connectivity patterns, which seems to affect the design I would recommend what has already been recommended - one or more centralized clusters that the devices feed into. You might also have local nodes that only ever are used for local analysis and also send the data to a centralized cluster. This duplicates the data but might give you a way to efficiently serve both modes. I have seen this kind of pattern work when you have isolated platforms with limitations on network connectivity and bandwidth, e.g. servers hosted on oil platforms etc. ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 11:48am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/32 "2019-04-02T11:48:56Z")
I understand your point. This has been suggested above as well. The disadvantages I see are 1. A constant flow of data even if the GUI is closed. 2. The central point is on the customer's site, not in the cloud. Therefore, I would have to request hardware enough for running a cluster capable to support all those millions of records, including constant updates. Also, I need to establish, configure and maintain the cluster. Not sure it's a difficult task, but it's not an insignificant one. 3. Since I have to deploy ES on each node anyway, why not to use my algorithm below? _From the "central" node query remote nodes. The result received from each node store on the central ES node Run the same query on the central ES node. In my opinion it should give the aggregated result across all the nodes._ ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 11:52am UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/33 "2019-04-02T11:52:56Z")
I suspect CCS will not scale to that number of clusters and even if it did I would expect you to experience very poor query performance as the slowest remote cluster potentially would dictate latency. If any cluster was temporarily unavailable you could also see errors and/or partial results. ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 12:12pm UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/34 "2019-04-02T12:12:04Z")
I totally agree with you. That's why I suggest the algorithm above. Is it a good approach? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 12:43pm UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/35 "2019-04-02T12:43:13Z")
How are you going to query the data? Are you going to use Kibana? As far as I can tell that approach would still suffer from the same drawbacks but also require s custom UI as Kibana does not support anything like that. ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 12:45pm UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/36 "2019-04-02T12:45:37Z")
I'm going to run HTTP requests via my Java-based server. Then I present the aggregated results on the GUI. No plans for Kibana right now. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 12:50pm UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/37 "2019-04-02T12:50:35Z")
I still think you are going to have a lot of problems with such approach and stick with my recommendation. Good luck! ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 12:51pm UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/38 "2019-04-02T12:51:25Z")
But could you clarify what kind of problems? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 1:47pm UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/39 "2019-04-02T13:47:36Z")
Given that it is an unusual approach I would not be surprised if you ran into the type of problems I described earlier plus some novel problems, limitations or edge cases that no one might have encountered before. ---
**Author:** ![vasekz](https://avatars.discourse-cdn.com/v4/letter/v/f475e1/32.png) [@vasekz](https://discuss.elastic.co/u/vasekz)\ **Post date:** [April 2, 2019, 1:48pm UTC](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959/40 "2019-04-02T13:48:29Z")
I still cannot figure out what problem could be with my algorithm. Can you be more specific? [Previous page](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959.md?page=1) [Next page](https://discuss.elastic.co/t/search-across-multiple-es-data-sources/171959.md?page=3)