Larger-than-memory mappings

I'm currently working on a project that will eventually have mappings
larger than a single server's available memory. User's have
the indirectly ability to create mappings so having the total size of the
mappings limited to a single server's available memory is inhibitive.

I've considered sharding per user (or groups of users), but this creates
some constraints that I'd prefer not to have (searching across shards isn't
preferable).

Am I worrying over nothing? Is the total size of the mapping bound by
single instance's available memory or disk space?

--

It sounds like you are indeed worrying a bit too early.
I assume you don't really mean that the mappings will be too big, but that
the index will be large?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, January 23, 2013 4:25:10 PM UTC-5, marcuslongmuir wrote:

I'm currently working on a project that will eventually have mappings
larger than a single server's available memory. User's have
the indirectly ability to create mappings so having the total size of the
mappings limited to a single server's available memory is inhibitive.

I've considered sharding per user (or groups of users), but this creates
some constraints that I'd prefer not to have (searching across shards isn't
preferable).

Am I worrying over nothing? Is the total size of the mapping bound by
single instance's available memory or disk space?

--

I do mean mappings. I know that the indexed data can exceed the size of a
single server, but I presume the assumption was made that mappings will
never grow larger than the size of single server's available memory.

I know that this isn't a common issue; it could only possibly occur if
users were indirectly creating mappings, which in my case they are.

On Thursday, January 24, 2013 5:00:36 AM UTC, Otis Gospodnetic wrote:

It sounds like you are indeed worrying a bit too early.
I assume you don't really mean that the mappings will be too big, but that
the index will be large?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, January 23, 2013 4:25:10 PM UTC-5, marcuslongmuir wrote:

I'm currently working on a project that will eventually have mappings
larger than a single server's available memory. User's have
the indirectly ability to create mappings so having the total size of the
mappings limited to a single server's available memory is inhibitive.

I've considered sharding per user (or groups of users), but this creates
some constraints that I'd prefer not to have (searching across shards isn't
preferable).

Am I worrying over nothing? Is the total size of the mapping bound by
single instance's available memory or disk space?

--

It will be an issue long before size of the mappings will reach the size of
a single server's available memory (unless you are running elasticsearch on
really tiny nodes). Mappings are part of cluster state and cluster state
is maintained by master and published to every non-master node on every
change.

On Thursday, January 24, 2013 12:04:28 AM UTC-5, marcuslongmuir wrote:

I do mean mappings. I know that the indexed data can exceed the size of a
single server, but I presume the assumption was made that mappings will
never grow larger than the size of single server's available memory.

I know that this isn't a common issue; it could only possibly occur if
users were indirectly creating mappings, which in my case they are.

On Thursday, January 24, 2013 5:00:36 AM UTC, Otis Gospodnetic wrote:

It sounds like you are indeed worrying a bit too early.
I assume you don't really mean that the mappings will be too big, but
that the index will be large?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, January 23, 2013 4:25:10 PM UTC-5, marcuslongmuir wrote:

I'm currently working on a project that will eventually have mappings
larger than a single server's available memory. User's have
the indirectly ability to create mappings so having the total size of the
mappings limited to a single server's available memory is inhibitive.

I've considered sharding per user (or groups of users), but this creates
some constraints that I'd prefer not to have (searching across shards isn't
preferable).

Am I worrying over nothing? Is the total size of the mapping bound by
single instance's available memory or disk space?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.