Caveats of a large mapping


(Nick Hoffman) #1

The documents that my application needs to index contain an ever-growing
number of arbitrary fields. Most of these fields are strings, but some are
dates or numbers. To be able to sort on these arbitrary fields, I've built
a dynamic template into my mapping:

As the number of documents and arbitrary fields grow, the dynamic template
will add fields to the mapping.

With only 1000 documents indexed, my mapping is 220 lines long. That's not
much, but after indexing 10,000 or 100,000 documents, that mapping will
grow significantly; probably upwards of 1,000 or 5,000 lines.

Are there any caveats that one should be aware of when doing this?

Thanks!


(Shay Banon) #2

The mappings are compressed and not "pretty printed" internally, so you
should not care about the size of it in that regard. Many fields do mean
more memory usage Lucene wise, just something to be aware.

On Thu, Nov 17, 2011 at 7:29 PM, Nick Hoffman nick@deadorange.com wrote:

The documents that my application needs to index contain an ever-growing
number of arbitrary fields. Most of these fields are strings, but some are
dates or numbers. To be able to sort on these arbitrary fields, I've built
a dynamic template into my mapping:
https://gist.github.com/e018df778fcece76717e

As the number of documents and arbitrary fields grow, the dynamic template
will add fields to the mapping.

With only 1000 documents indexed, my mapping is 220 lines long. That's not
much, but after indexing 10,000 or 100,000 documents, that mapping will
grow significantly; probably upwards of 1,000 or 5,000 lines.

Are there any caveats that one should be aware of when doing this?

Thanks!


(Nick Hoffman) #3

Thanks for the clarification!


(system) #4