Json documents with large number of fields

Elastic Stack Elasticsearch
1

Hi all,

I have around 20K JSON documents that have around 350 fields to be pushed
into ES (0.90.13). The mapping type is all multi valued string fields.
Currently I have a single shard and 0 replicas.

My question is whether ES(Lucene) is capable of handling these large
documents? Or does it work well for small documents (less fields) but
larger number of such small documents?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

This is not large. I have 5500 fields in ~100m docs, most fields are not
analyzed / not indexed. Sure, this works perfectly.

Jörg

On Sun, Nov 23, 2014 at 10:19 AM, Ajay Divakaran <ajay.divakaran86@gmail.com

wrote:

Hi all,

I have around 20K JSON documents that have around 350 fields to be pushed
into ES (0.90.13). The mapping type is all multi valued string fields.
Currently I have a single shard and 0 replicas.

My question is whether ES(Lucene) is capable of handling these large
documents? Or does it work well for small documents (less fields) but
larger number of such small documents?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGoO1jaUYV%2BM3adsFA%2BTfH5yXRG1r7ZMMTsKsdW%3DwN9NQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Jorg,

Thanks for your reply.

Does the memory spike when you retrieve these documents in bulk as a result
of a query/filter result?

On Sunday, November 23, 2014 7:43:49 PM UTC+5:30, Jörg Prante wrote:

This is not large. I have 5500 fields in ~100m docs, most fields are not
analyzed / not indexed. Sure, this works perfectly.

Jörg

On Sun, Nov 23, 2014 at 10:19 AM, Ajay Divakaran <ajay.div...@gmail.com
<javascript:>> wrote:

Hi all,

I have around 20K JSON documents that have around 350 fields to be pushed
into ES (0.90.13). The mapping type is all multi valued string fields.
Currently I have a single shard and 0 replicas.

My question is whether ES(Lucene) is capable of handling these large
documents? Or does it work well for small documents (less fields) but
larger number of such small documents?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00b46a98-4d64-40d9-a011-b8b7d29dcaa6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

No.

Use filtered query to avoid memory spikes by post filter.

Jörg

On Sun, Nov 23, 2014 at 6:22 PM, Ajay Divakaran ajay.divakaran86@gmail.com
wrote:

Jorg,

Thanks for your reply.

Does the memory spike when you retrieve these documents in bulk as a
result of a query/filter result?

On Sunday, November 23, 2014 7:43:49 PM UTC+5:30, Jörg Prante wrote:

This is not large. I have 5500 fields in ~100m docs, most fields are not
analyzed / not indexed. Sure, this works perfectly.

Jörg

On Sun, Nov 23, 2014 at 10:19 AM, Ajay Divakaran ajay.div...@gmail.com
wrote:

Hi all,

I have around 20K JSON documents that have around 350 fields to be
pushed into ES (0.90.13). The mapping type is all multi valued string
fields.
Currently I have a single shard and 0 replicas.

My question is whether ES(Lucene) is capable of handling these large
documents? Or does it work well for small documents (less fields) but
larger number of such small documents?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d746894-2e54-4215-a489-bda5708515f1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/00b46a98-4d64-40d9-a011-b8b7d29dcaa6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/00b46a98-4d64-40d9-a011-b8b7d29dcaa6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE9cLxhNgf0-PX4xBZ%3D_ZPx8%3DkFSumhCcW9y%2Bx_kJgSyA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.