Stored Field with Disabled Source

Elastic Stack Elasticsearch
1

Hi,

Just wanted to check - I'm currently trying to run an index with disabled
source (to improve storage/speed since we have an additional data source)
but we wish to keep certain fields stored (like a preview text).

I'm currently running my index with the settings below, but returned search
results does not seem to provided "stored" preview field.

Is this possible? I'm running Elasticsearch 0.20.1

Thanks!

Joshua

"mappings" : {
"chat" : {
"_source" : {
"compress": True, # Not needed but could index be compressed?
"enabled" : False
},
"properties" : {
"text": {
"type": "string"
},
"preview": {
"type": "string",
"store": "yes"
}
}
}
}

--

2

Hello!

Have you tried forcing returning the 'preview' field with fields params - http://www.elasticsearch.org/guide/reference/api/search/fields.html ?

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch

Hi,

Just wanted to check - I'm currently trying to run an index with disabled source (to improve storage/speed since we have an additional data source) but we wish to keep certain fields stored (like a preview text).

I'm currently running my index with the settings below, but returned search results does not seem to provided "stored" preview field.

Is this possible? I'm running Elasticsearch 0.20.1

Thanks!

Joshua

"mappings" : {

"chat" : {

"_source" : {

"compress": True, # Not needed but could index be compressed?

"enabled" : False

},

"properties" : {

"text": {

"type": "string"

},

"preview": {

"type": "string",


"store": "yes"

}

}

}

}

--

3

Thanks for the pointer, Rafal!

Running a query like this would work!

{
"fields" : ["*"],
"query" : {
"match_all":{}
}
}

If I may ask another question - is there an option for the stored field and
main index to be compressed as well?

Cheers,
Joshua

On Saturday, December 22, 2012 6:39:40 PM UTC+8, Rafał Kuć wrote:

Hello!

Have you tried forcing returning the 'preview' field with fields params -
Elasticsearch Platform — Find real-time answers at scale | Elastic ?

*--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
Elasticsearch

Hi,

Just wanted to check - I'm currently trying to run an index with disabled
source (to improve storage/speed since we have an additional data source)
but we wish to keep certain fields stored (like a preview text).

I'm currently running my index with the settings below, but returned
search results does not seem to provided "stored" preview field.

Is this possible? I'm running Elasticsearch 0.20.1

Thanks!

Joshua

"mappings" : {
"chat" : {
"_source" : {
"compress": True, # Not needed but could index be compressed?
"enabled" : False
},
"properties" : {
"text": {
"type": "string"
},
"preview": {
"type": "string",
"store": "yes"
}
}
}
}

--

4

You can enable store level compression of all fields with

index.store.compress.stored: true

in the configuration.

http://www.elasticsearch.org/guide/reference/index-modules/store.html

Jörg

--

5

Thanks Jörg,

Somehow, compression on all fields does not seem to have any visible size
benefits (in fact, index.store.compress.tv seems to take up more
space) when source store is disabled.

I'm running some tests of 126434 documents, here are the results of
indexing it with different settings (using 1 shard, 0 replicas).

Defaults: 499.6MB
_source: { enabled: True, compress: False }

Defaults + Compression: 207.8MB
_source: { enabled: True, compress: True }

Stored source + "index.store.compress.stored": 162.7MB
settings: { "index.store.compress.stored" : True }

Stored source + Compressed Term Vector: 180.2MB
settings: { "index.store.compress.stored" : True, "index.store.compress.tv":
True }
*
*

This seems to be correct, but when source is disabled with
_source: { enabled: False }

here are the results:

Source store disabled: 120.5MB

Source store disabled + Compressed Term Vector: 121MB

Source store disabled + "index.store.compress.stored": 120.9MB

Does this mean that the index are not really compressed at all or could it
be my configurations are wrongly configured?

Cheers,
Joshua

On Mon, Dec 24, 2012 at 10:03 PM, Jörg Prante joergprante@gmail.com wrote:

You can enable store level compression of all fields with

index.store.compress.stored: true

in the configuration.

Elasticsearch Platform — Find real-time answers at scale | Elastic

Jörg

--

--

6

Hi,

please disregard my last mail if it was misleading - I reran my tests and I
found some interesting results -

whether index.store.compress.stored or *index.store.compress.tv *is set
to true,

the compress field below does make a difference to the index size, even
when source is not enabled.

_source: { compress: True, enabled: False }
*
*
Take for example
*
*
_source: { compress: False, enabled: False }
*creates *183.8MB
*
*
*_source: { compress: False, enabled: False }, **
*index.store.compress.stored:
True
*creates *122.1MB
*
*
*_source: { compress: True, enabled: False }, **
*index.store.compress.stored:
True
*creates *112.6MB
*
*
*_source: { enabled: False }, *index.store.compress.stored: True, **
index.store.compress.tv: True
*creates *125.6MB
*
*
*_source: { compress: True, enabled: False },
**index.store.compress.stored:
True, *index.store.compress.tv: True
*creates *111.9MB
*
*
My conclusion is that for maximum space savings - one should have have
_source disabled, _source compression enabled, and
*index.store.compress.stored
*enabled.
*
*
Thanks!
*
*
Cheers,
Joshua

On Wed, Dec 26, 2012 at 1:07 PM, Joshua Koo joshua@zopim.com wrote:

Thanks Jörg,

Somehow, compression on all fields does not seem to have any visible size
benefits (in fact, index.store.compress.tv seems to take up more
space) when source store is disabled.

I'm running some tests of 126434 documents, here are the results of
indexing it with different settings (using 1 shard, 0 replicas).

Defaults: 499.6MB
_source: { enabled: True, compress: False }

Defaults + Compression: 207.8MB
_source: { enabled: True, compress: True }

Stored source + "index.store.compress.stored": 162.7MB
settings: { "index.store.compress.stored" : True }

Stored source + Compressed Term Vector: 180.2MB
settings: { "index.store.compress.stored" : True, "
index.store.compress.tv": True }
*
*

This seems to be correct, but when source is disabled with
_source: { enabled: False }

here are the results:

Source store disabled: 120.5MB

Source store disabled + Compressed Term Vector: 121MB

Source store disabled + "index.store.compress.stored": 120.9MB

Does this mean that the index are not really compressed at all or could it
be my configurations are wrongly configured?

Cheers,
Joshua

On Mon, Dec 24, 2012 at 10:03 PM, Jörg Prante joergprante@gmail.comwrote:

You can enable store level compression of all fields with

index.store.compress.stored: true

in the configuration.

Elasticsearch Platform — Find real-time answers at scale | Elastic

Jörg

--

--