# Elasticsearch ingest large of data **URL:** \ **Category:** Elasticsearch\ **Created:** [December 28, 2022, 1:04pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-large-of-data/322099 "2022-12-28T13:04:08Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![m3bgwad](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@m3bgwad](https://discuss.elastic.co/u/m3bgwad)\ **Post date:** [December 28, 2022, 1:04pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-large-of-data/322099/1 "2022-12-28T13:04:08Z")
is there capability of Elasticsearch to ingest amount large of data at a rate of 300TB in the year after that archive data in offline storage? what is challenges facing me to ingest that much volume? ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [December 28, 2022, 3:10pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-large-of-data/322099/2 "2022-12-28T15:10:12Z")
I would say that Elasticsearch should be able to handle this data rate, but I'd recommend looking into a few key concepts: 1. 300TB is a lot of data, having that all on one cluster would probably require a fairly large cluster. If possible, try to break the data into logical groupings and have an Elasticsearch cluster for each logical group. You can then use [cross cluster searching](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html) to have a single "search" cluster, search all of your logical group clusters. 2. At this scale, you'd most likely want dedicated [ingest](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-ingest-node) and [coordinating](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#coordinating-only-node) nodes. 3. If possible, look into using Elasticsearch's [data tiering system](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tiers.html). This will allow for greater data storage density with lower costs if possible. 4. At this scale, you'll want to make sure your mappings and ingest pipelines are as efficient as possible (and thoroughly tested) to ensure reliable performance. 5. Ensure you're on the latest version of Elasticsearch. Scalability has gotten significantly better in the last few versions (and is continuing to improve), being able to take advantage of these improvements will probably make this far more pleasant. 6. No matter how you look at it, you will need some powerful systems (CPU, RAM, Disk, Network) to be able to support this type of data rates. I would recommend you someone try to do some sort of semi-scale load testing to ensure you're able to handle this. 7. Somewhat related to #6, ensure your backup solution is able to handle backing up the data fast enough. During the backup process, there are some other things that don't work (example: Indices can't rollover during a backup, if the index is included in the backup). Ensuring that your backups finish in a timely manner will reduce other potential issues that might come up as a result of them taking a while to run. I'm sure I'm missing some other considerations, but others here might be able to provide more insight As a case of theory, using [https://cloud.elastic.co/pricing](https://cloud.elastic.co/pricing), you can build a cluster (using all data tiers) that supports ~11.5PB of data total. (Note that this is mainly just for theory, a single cluster this large, while probably do-able, probably isn't the best idea; See point #1) ---
**Author:** ![m3bgwad](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@m3bgwad](https://discuss.elastic.co/u/m3bgwad)\ **Post date:** [December 29, 2022, 7:45am UTC](https://discuss.elastic.co/t/elasticsearch-ingest-large-of-data/322099/3 "2022-12-29T07:45:06Z")
Thank You For sharing, regarding this point, > [@BenB196](#): > > As a case of theory, using [https://cloud.elastic.co/pricing](https://cloud.elastic.co/pricing), you can build a cluster (using all data tiers) that supports ~11.5PB of data total. (Note that this is mainly just for theory, a single cluster this large, while probably do-able, probably isn't the best idea; See point #1) If I Need achieve this capability internally and I will provide the powerful of system like ( CPU, RAM, Disk, Network ) and I will take that into consideration index and search speed tuning . > [@BenB196](#): > > 300TB is a lot of data, having that all on one cluster would probably require a fairly large cluster. If possible, try to break the data into logical groupings and have an Elasticsearch cluster for each logical group. You can then use [cross cluster searching](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html) to have a single In your epinine what is differences between cross cluster searching and single cluster searching, you mean the cross cluster searching will be provide the high performance in side the searching and index speed? what will the provide for the logical group clusters? ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [December 29, 2022, 12:59pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-large-of-data/322099/4 "2022-12-29T12:59:47Z")
> In your opinion what is differences between cross cluster searching and single cluster searching, you mean the cross-cluster searching will be provide the high performance inside the searching and index speed? There are a few pros and cons to each architecture and not all are strictly performance related. **Cross-cluster Search** Pros: - Smaller clusters - When Elasticsearch makes a change, it populates that change across all nodes, the more nodes in a cluster, the longer it takes for that change to propagate, slowing down things. - Note: I recall there being a post somewhere about this, but couldn't find the link, if this info is incorrect, or if someone has the link, please let me know. - Shorter upgrade durations. - On average you can assume that it takes ~15-20 to upgrade an Elasticsearch node. With this in mind, the more nodes in a cluster, the longer it will take to perform a full (rolling) upgrade. Smaller clusters allow for doing one cluster at a time in a quicker interval. - Upgrade validation - While you should always be testing new versions of Elasticsearch in some sort of test environment, no environment will ever truly match production. By being able to upgrade only part of your production infrastructure at a time, you have a better chance of detecting issues and "fixing" them, without impacting your entire production environment. - Targeted performance - If you know what data lives on a specific cluster in a cross-cluster search setup, you can more specifically target the resources of that cluster to obtain the desired performance for that data. - Requires you to understand your data better - To be able to setup a viable cross-cluster search architecture, you must understand your data well enough to know how it will be used and what it is. Without this knowledge you probably would have a great cross-cluster search architecture - There is a secondary advantage here, if you understand your data better, you can get improvements elsewhere like index mappings. - Scaling - While I'm pretty sure a single Elasticsearch cluster can now scale to petabytes of data and 10's of thousands of indices, there is probably some eventual limitation out there. When using cross-cluster this limitation effectively goes away as you can probably have a large number of clusters as part of the cross-cluster search setup. Cons: - More clusters - With Cross-cluster search, you need to deal with multiple clusters, which generally adds "management" overhead in the form of additional configuration for each cluster, upgrading, and monitoring them. This adds to the overall "total cost of ownership", and things to worry about - More complex Architecture, similar to the above point, cross-cluster search does require a more complex architectural setup than its alternative, a single cluster. - Requires you to understand your data - While this is a pro, it could also be considered a con. If you currently don't understand your data and how it will be used, trying to gain this knowledge to be able to properly implement cross-cluster search can take a significant amount of time if your data is large and covers a wide range of "areas". - Feature parity - There are some features currently in Elasticsearch that don't support/work well with cross-cluster search. It's important to know what features you want to use and validate that they work with it. - Note: Elasticsearch has been making recent enhancements and many more features now support cross-cluster search. **Single Elasticsearch Cluster** Pros - Relatively simple architecture - Don't really need to fully understand your data - Feature support - Lower total cost of ownership - Relatively Limited scaling - Un-targetable performance - Long upgrades - Potentially lower performance Note: While this is mainly comparing cross-cluster search and a single cluster. It is important to note, that in theory you can start with one of the architectures and transition to the other if your use-case changes. (It will require additional work to rearchitect though) > what will the provide for the logical group clusters? I don't really understand this question. ---
**Author:** ![m3bgwad](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@m3bgwad](https://discuss.elastic.co/u/m3bgwad)\ **Post date:** [December 29, 2022, 1:21pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-large-of-data/322099/5 "2022-12-29T13:21:07Z")
I mean What will provide the logical group clusters? what is addition will it give the logical group clusters to improve the performance? ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [December 29, 2022, 1:24pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-large-of-data/322099/6 "2022-12-29T13:24:47Z")
> What will provide the logical group clusters? This requires you to know and understand your data and how it can be segmented into "logical" groups that could become their own Elasticsearch cluster as part of a cross-cluster search architecture. > what is addition will it give the logical group clusters to improve the performance? See the pros of **Cross-cluster Search** on my last response. (Not everything is about raw performance are larger scales) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 26, 2023, 1:24pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-large-of-data/322099/7 "2023-01-26T13:24:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.