# How to index a large dataset **URL:** \ **Category:** Elasticsearch\ **Created:** [August 2, 2017, 1:00pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526 "2017-08-02T13:00:25Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Erates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erates/32/9406_2.png) [@Erates](https://discuss.elastic.co/u/Erates)\ **Post date:** [August 2, 2017, 1:00pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526/1 "2017-08-02T13:00:25Z")
Hello, At the moment, I have a dataset containing over 170 million records and I want to index it in Elasticsearch. But the problem is that the way I need the mapping to be, generates to much datastorage. Our data looks like this: in\_category (max 256 characters) in\_filename (max 256 characters) in\_timestamp out\_category (max 256 characters) out\_filename (max 256 characters) out\_timestamp Some other fields are there as well, but they are not so dynamic. The mapping has to allow for searches where the category (in\_ or out\_) has to start with a prefix, contain some text, ends with a postfix or an exact match. Also, for the filename (in\_ and out\_) we need to be able to search for prefixes, postfixes, data within the filename and an exact match. So for example, let's take a filename "my\_file\_20170802-1454.pdf". I need to be able to find this when I enter one of the following queries: - my\_file - 20170802 - pdf - 2017 - my\_file\_20170802-1454.pdf Basically for the category, the same is true, only other datastructure. Usually containing underscores, hypens or spaces. I have configured a mapping which allows this behavior, but it'll take up to 1,5TB of storage. Is there a simpler way to handle this? Erates ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 2, 2017, 1:23pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526/2 "2017-08-02T13:23:17Z")
IMO the way to solve this is by either using ngrams (size of 3 as in your example the lowest number of characters is 3: `pdf`). But may be using a [pattern tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-tokenizer.html) would help to produce the exact terms you mentioned? FWIW, ngrams or pattern tokenizers will use mode disk space for sure. ---
**Author:** ![Erates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erates/32/9406_2.png) [@Erates](https://discuss.elastic.co/u/Erates)\ **Post date:** [August 2, 2017, 1:35pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526/3 "2017-08-02T13:35:38Z")
Thanks for the reply! In the mapping I currently have, I'm using ngrams, 3-25, and another field for exact matching. And this is giving us a storage (calculated) of over 1,5TB. IMO, 500GB would be acceptable, but 1,5TB is a bit much. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 2, 2017, 2:34pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526/4 "2017-08-02T14:34:52Z")
Well. I'm not sure what you can do. I think it's a tradeoff between index space and speed at query time. If you don't want to pay the space price, you'll have to pay the speed price by running super slow wildcard queries. BTW did you disable the `_all` field? ---
**Author:** ![Erates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erates/32/9406_2.png) [@Erates](https://discuss.elastic.co/u/Erates)\ **Post date:** [August 2, 2017, 3:03pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526/5 "2017-08-02T15:03:24Z")
No we did not. I'll run another test with the field disabled. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 2, 2017, 3:15pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526/6 "2017-08-02T15:15:42Z")
Not sure about what you tested but can you compare just with adding a ngram based field with no field added? In term of default space. ---
**Author:** ![Erates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erates/32/9406_2.png) [@Erates](https://discuss.elastic.co/u/Erates)\ **Post date:** [August 2, 2017, 3:24pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526/7 "2017-08-02T15:24:13Z")
Well, I took a subset of the data and ran it into a test elastic search instance. Then after that was done, I let the index optimize itself (because from experience, I've noticed that just pushing it in and then checking disk size is not reliable). Then with that data I've extrapolated until we had a "guess" about what the size would be with the full dataset. That test already uses the n-gram filter, 2-20 for category and 3-25 for the filename. Both 2 times (in\_ and out\_). I still have to run the test without the `_all` field. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 2, 2017, 4:23pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526/8 "2017-08-02T16:23:56Z")
You can call the `forcemerge` API if needed so you will be able to compare somehow with the same number of segments. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 30, 2017, 4:24pm UTC](https://discuss.elastic.co/t/how-to-index-a-large-dataset/95526/9 "2017-08-30T16:24:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.