# Thousands of fields in a mapping type?
**URL:** https://discuss.elastic.co/t/thousands-of-fields-in-a-mapping-type/7576
**Category:** Elasticsearch
**Created:** [May 6, 2012, 6:53pm UTC](https://discuss.elastic.co/t/thousands-of-fields-in-a-mapping-type/7576 "2012-05-06T18:53:02Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@aaron](https://discuss.elastic.co/u/aaron)
#### Post date: [May 6, 2012, 6:53pm UTC](https://discuss.elastic.co/t/thousands-of-fields-in-a-mapping-type/7576/1 "2012-05-06T18:53:02Z")
Would it be a bad idea to define a universal type containing thousands,
perhaps tens of thousands, of fields within a single elasticsearch Mapping?
The documents I store for a given index would use a subset of the fields in
the mapping but for management reasons it would be convenient to have a
universal mapping type that I could use for all indexes.
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [May 7, 2012, 12:36am UTC](https://discuss.elastic.co/t/thousands-of-fields-in-a-mapping-type/7576/2 "2012-05-07T00:36:18Z")
I have never explored the impact on performance with thousands of
types (not even hundreds), but you can override the default behavior
of types or use dynamic templates to define a universal type:
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
It depends of you want to override the defaults based on type (string,
boolean, etc...) or field name. I believe I have read at some point
that too many types does have an impact on performance.
--
Ivan
On Sun, May 6, 2012 at 11:53 AM, aaron [atdixon@gmail.com](mailto:atdixon@gmail.com) wrote:
> Would it be a bad idea to define a universal type containing thousands,
> perhaps tens of thousands, of fields within a single elasticsearch Mapping?
>
> The documents I store for a given index would use a subset of the fields in
> the mapping but for management reasons it would be convenient to have a
> universal mapping type that I could use for all indexes.
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 9, 2012, 8:55am UTC](https://discuss.elastic.co/t/thousands-of-fields-in-a-mapping-type/7576/3 "2012-05-09T08:55:32Z")
Each field comes with an overhead of memory usage (on the Lucene level),
theoretically its possible, but you will need to check the memory (sadly,
the memory used by Lucene is not exposed to be reported by ES).
On Sun, May 6, 2012 at 9:53 PM, aaron [atdixon@gmail.com](mailto:atdixon@gmail.com) wrote:
> Would it be a bad idea to define a universal type containing thousands,
> perhaps tens of thousands, of fields within a single elasticsearch Mapping?
>
> The documents I store for a given index would use a subset of the fields
> in the mapping but for management reasons it would be convenient to have a
> universal mapping type that I could use for all indexes.
---
### Author:  [@Bruce\_Lysik](https://discuss.elastic.co/u/Bruce_Lysik)
#### Post date: [February 11, 2013, 5:22pm UTC](https://discuss.elastic.co/t/thousands-of-fields-in-a-mapping-type/7576/4 "2013-02-11T17:22:40Z")
On Wednesday, May 9, 2012 1:55:32 AM UTC-7, kimchy wrote:
> Each field comes with an overhead of memory usage (on the Lucene level),
> theoretically its possible, but you will need to check the memory (sadly,
> the memory used by Lucene is not exposed to be reported by ES).
How many fields is excessive? Currently I'm storing syslog and apache
logs, but I'm considering giving the development teams to log anything in
json format, which would let them create any fields they please.
Is 100 fields too many? Or 1000? Any guidance would be appreciated.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)
#### Post date: [February 11, 2013, 8:21pm UTC](https://discuss.elastic.co/t/thousands-of-fields-in-a-mapping-type/7576/5 "2013-02-11T20:21:07Z")
Hello Bruce,
Nice to meet you again 🙂 Although you got some of my opinion on the
subject on the Logstash ML, here's a second shot:
On Mon, Feb 11, 2013 at 7:22 PM, Bruce Lysik [blysik@yahoo.com](mailto:blysik@yahoo.com) wrote:
> On Wednesday, May 9, 2012 1:55:32 AM UTC-7, kimchy wrote:
>
> > Each field comes with an overhead of memory usage (on the Lucene level),
> > theoretically its possible, but you will need to check the memory (sadly,
> > the memory used by Lucene is not exposed to be reported by ES).
>
> How many fields is excessive? Currently I'm storing syslog and apache
> logs, but I'm considering giving the development teams to log anything in
> json format, which would let them create any fields they please.
In production you probably want to have control over those fields, instead
of relying on ES to detect field types for you. One reason for that is if
the first log in an index accidentally contains an integer in a field that
would normally be string, you'll get indexing errors for pretty much all
the other logs that day.
How many fields do you expect to get in total?
> Is 100 fields too many? Or 1000? Any guidance would be appreciated.
Maybe someone else can pop in with some benchmarks, but the sound of 100
and 1000 fields doesn't seem too many to me.
Of course, the definition of excessive will depend on quite some factors,
like:
- how much memory you have
- how many mapping types you have per index (you'll have to add them up to
get the total number of fields per Lucene index)
- what search performance you're expecting (which in turn depends on how
much data you have in an index, how many indices, shards...)
So if you want to make sure, you can run a performance test with your
worst-case scenario and see if all goes well. I'd also recommend monitoring
your cluster during the test, to see what limits you're approaching. There
are quite a lot of nice tools out there for monitoring ES, one of which is
ours:
> **[Elasticsearch - Sematext Documentation](https://sematext.com/docs/integration/elasticsearch-integration/)**
>
> Collect and monitor key Elasticsearch metrics such as request latency, indexing rate, and segment merges with built-in anomaly detection, threshold, and heartbeat alerts. Send notifications to email and various chatops messaging services, correlate...
## Best regards, Radu
[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:52am UTC](https://discuss.elastic.co/t/thousands-of-fields-in-a-mapping-type/7576/6 "2017-07-06T02:52:00Z")