# Compression flag is not wrking **URL:** \ **Category:** Elasticsearch\ **Created:** [April 20, 2012, 10:49pm UTC](https://discuss.elastic.co/t/compression-flag-is-not-wrking/7428 "2012-04-20T22:49:55Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Anita](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@Anita](https://discuss.elastic.co/u/Anita)\ **Post date:** [April 20, 2012, 10:49pm UTC](https://discuss.elastic.co/t/compression-flag-is-not-wrking/7428/1 "2012-04-20T22:49:55Z")
Compression flag is not working. ElasticSearch version : 0.19.1 Gateway : Hadoop Hadoop version: 0.20.2.cdh3u3 We are planning to use ElasticSearch and Hadoop gateway to store and search log data. There will be few hundred tera bytes of log data. TEST1: Index: order INDEX STATUS { state: open settings: { index.number\_of\_shards: 10 index.number\_of\_replicas: 0 index.version.created: 190199 index.\_source.compress: true } Compression flag is set by running curl request curl -XPUT localhost:9200/order/\_settings -d '{"index": {"\_source" : {"compress" : true} }}' I ran the optimizer curl -XPOST '[http://localhost:9200/order/\_optimize?max\_num\_segments=5](http://localhost:9200/order/_optimize?max_num_segments=5)' Order size: 1.2gb (1.2gb) docs: 349345 (349345) Each log message is about 1K. I am writing raw data to a file to check how compression works Raw data : 381978518 Apr 20 12:40 test\_es.log ( Non compressed 380M) Data written on the hdfs : 1304717764 bytes (1.2G) Raw data size is about 380M, how come elasticsearch is written about 1.2 G of data even with compress flag is enabled. ElasticSearch wrote about 3 times of raw data. I do not see data getting compressed at all. In addition to log message fields, elasticsearch writes few more fields, but how come elasticsearch data size 3 times of original data size? Am I missing any thing here? Each document contains \_index: order \_type: trade \_id: 5c10PkonSeOQrBZkH8JVpg \_version: 1 \_score: 1 \_source: { ``` // source cntains log message. It has 12 fields. Log message is about 1K. ``` } TEST2: I tested with/without compression. Cluster : 3 node cluster Replication : 2 Number of shards : 10 Gateway : hadoop (5 node cluster, replication 3) Compression Test INDEX META DATA { state: open settings: { index.number\_of\_shards: 10 index.number\_of\_replicas: 2 index.version.created: 190199 index.\_source.compress: true } index.\_source.compress: true order size: 2.2gb (6.7gb) docs: 627336 (627336) Same log data writing to a file: 685678405 Apr 20 10:44 test\_es.log ( size 685M) Raw data is only about 685M Without Compression { order size: 2.2gb (6.8gb) docs: 631361 (631361) 690077730 Apr 20 12:45 test\_es.log (SIZE 690M) I do not see any difference in data size w/o compression, when I run "du -sh" on the data directory. is it right way to set compression flag? curl -XPUT localhost:9200/order/\_settings -d '{"index": {"\_source" : {"compress" : true} }}' curl -XPUT localhost:9200/order/\_settings -d '{"index": {"\_source" : {"compress\_threshold" : 10000} }} I really appreciate any help on this. Please give me some guidelines how to scale large data. ---
**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\ **Post date:** [April 25, 2012, 12:24pm UTC](https://discuss.elastic.co/t/compression-flag-is-not-wrking/7428/2 "2012-04-25T12:24:31Z")
Do you have any fields included in the \_source? Compression does not happen on the index overall, just those fields included in the \_source field. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. The \_source field is an automatically generated field that stores the actual JSON that was used as the indexed document. ... Includes / Excludes Allow to specify paths in the source that would be included / excluded when its stored, supporting \* as wildcard annotation. On Fri, Apr 20, 2012 at 6:49 PM, Anita [narayana\_anita@hotmail.com](mailto:narayana_anita@hotmail.com) wrote: > Compression flag is not working. > > Elasticsearch version : 0.19.1 > Gateway : Hadoop > Hadoop version: 0.20.2.cdh3u3 > > We are planning to use Elasticsearch and Hadoop gateway to store and search > log data. > There will be few hundred tera bytes of log data. > > TEST1: > Index: order > > INDEX STATUS > { > state: open > settings: { > index.number\_of\_shards: 10 > index.number\_of\_replicas: 0 > index.version.created: 190199 > index.\_source.compress: true > } > Compression flag is set by running curl request > curl -XPUT localhost:9200/order/\_settings -d '{"index": {"\_source" : > {"compress" : true} }}' > I ran the optimizer > curl -XPOST '[http://localhost:9200/order/\_optimize?max\_num\_segments=5](http://localhost:9200/order/_optimize?max_num_segments=5)' > > Order > size: 1.2gb (1.2gb) > docs: 349345 (349345) > > Each log message is about 1K. > > I am writing raw data to a file to check how compression works > Raw data : 381978518 Apr 20 12:40 test\_es.log ( Non compressed 380M) > > Data written on the hdfs : 1304717764 bytes (1.2G) > > Raw data size is about 380M, how come elasticsearch is written about 1.2 G > of data even with compress flag is enabled. > Elasticsearch wrote about 3 times of raw data. > I do not see data getting compressed at all. In addition to log message > fields, elasticsearch writes few more fields, but how come elasticsearch > data size 3 times of original data size? Am I missing any thing here? > > Each document contains > > \_index: order > \_type: trade > \_id: 5c10PkonSeOQrBZkH8JVpg > \_version: 1 > \_score: 1 > \_source: { > > ``` > // source cntains log message. It has 12 fields. Log message is about > > ``` > > 1K. > > } > > TEST2: > I tested with/without compression. > > Cluster : 3 node cluster > Replication : 2 > Number of shards : 10 > Gateway : hadoop (5 node cluster, replication 3) > > Compression Test > > INDEX META DATA > { > state: open > settings: { > index.number\_of\_shards: 10 > index.number\_of\_replicas: 2 > index.version.created: 190199 > index.\_source.compress: true > } > > index.\_source.compress: true > > order > size: 2.2gb (6.7gb) > docs: 627336 (627336) > > Same log data writing to a file: > > 685678405 Apr 20 10:44 test\_es.log ( size 685M) > > Raw data is only about 685M > > Without Compression > { > order > size: 2.2gb (6.8gb) > docs: 631361 (631361) > > 690077730 Apr 20 12:45 test\_es.log (SIZE 690M) > > I do not see any difference in data size w/o compression, when I run "du > -sh" on the data directory. > > is it right way to set compression flag? > curl -XPUT localhost:9200/order/\_settings -d '{"index": {"\_source" : > {"compress" : true} }}' > curl -XPUT localhost:9200/order/\_settings -d '{"index": {"\_source" : > {"compress\_threshold" : 10000} }} > > I really appreciate any help on this. > > Please give me some guidelines how to scale large data. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/compression-flag-is-not-wrking-tp3927247p3927247.html](http://elasticsearch-users.115913.n3.nabble.com/compression-flag-is-not-wrking-tp3927247p3927247.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [April 25, 2012, 2:51pm UTC](https://discuss.elastic.co/t/compression-flag-is-not-wrking/7428/3 "2012-04-25T14:51:30Z")
\_source is a field, so it should configured in the mapping like this: curl -XPUT [http://localhost:9200/order/trade/\_mapping](http://localhost:9200/order/trade/_mapping) -d '{ "trade" : { "\_source" : { "compress" : true } } }' On Wednesday, April 25, 2012 8:24:31 AM UTC-4, Michael Sick wrote: > Do you have any fields included in the \_source? Compression does not > happen on the index overall, just those fields included in the \_source > field. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/source-field.html) > The \_source field is an automatically generated field that stores the > actual JSON that was used as the indexed document. > > ... > > Includes / Excludes > > Allow to specify paths in the source that would be included / excluded > when its stored, supporting \* as wildcard annotation. > > On Fri, Apr 20, 2012 at 6:49 PM, Anita [narayana\_anita@hotmail.com](mailto:narayana_anita@hotmail.com) wrote: > > > Compression flag is not working. > > > > Elasticsearch version : 0.19.1 > > Gateway : Hadoop > > Hadoop version: 0.20.2.cdh3u3 > > > > We are planning to use Elasticsearch and Hadoop gateway to store and > > search > > log data. > > There will be few hundred tera bytes of log data. > > > > TEST1: > > Index: order > > > > INDEX STATUS > > { > > state: open > > settings: { > > index.number\_of\_shards: 10 > > index.number\_of\_replicas: 0 > > index.version.created: 190199 > > index.\_source.compress: true > > } > > Compression flag is set by running curl request > > curl -XPUT localhost:9200/order/\_settings -d '{"index": {"\_source" : > > {"compress" : true} }}' > > I ran the optimizer > > curl -XPOST '[http://localhost:9200/order/\_optimize?max\_num\_segments=5](http://localhost:9200/order/_optimize?max_num_segments=5)' > > > > Order > > size: 1.2gb (1.2gb) > > docs: 349345 (349345) > > > > Each log message is about 1K. > > > > I am writing raw data to a file to check how compression works > > Raw data : 381978518 Apr 20 12:40 test\_es.log ( Non compressed 380M) > > > > Data written on the hdfs : 1304717764 bytes (1.2G) > > > > Raw data size is about 380M, how come elasticsearch is written about 1.2 G > > of data even with compress flag is enabled. > > Elasticsearch wrote about 3 times of raw data. > > I do not see data getting compressed at all. In addition to log message > > fields, elasticsearch writes few more fields, but how come elasticsearch > > data size 3 times of original data size? Am I missing any thing here? > > > > Each document contains > > > > \_index: order > > \_type: trade > > \_id: 5c10PkonSeOQrBZkH8JVpg > > \_version: 1 > > \_score: 1 > > \_source: { > > > > ``` > > // source cntains log message. It has 12 fields. Log message is about > > > > ``` > > > > 1K. > > > > } > > > > TEST2: > > I tested with/without compression. > > > > Cluster : 3 node cluster > > Replication : 2 > > Number of shards : 10 > > Gateway : hadoop (5 node cluster, replication 3) > > > > Compression Test > > > > INDEX META DATA > > { > > state: open > > settings: { > > index.number\_of\_shards: 10 > > index.number\_of\_replicas: 2 > > index.version.created: 190199 > > index.\_source.compress: true > > } > > > > index.\_source.compress: true > > > > order > > size: 2.2gb (6.7gb) > > docs: 627336 (627336) > > > > Same log data writing to a file: > > > > 685678405 Apr 20 10:44 test\_es.log ( size 685M) > > > > Raw data is only about 685M > > > > Without Compression > > { > > order > > size: 2.2gb (6.8gb) > > docs: 631361 (631361) > > > > 690077730 Apr 20 12:45 test\_es.log (SIZE 690M) > > > > I do not see any difference in data size w/o compression, when I run "du > > -sh" on the data directory. > > > > is it right way to set compression flag? > > curl -XPUT localhost:9200/order/\_settings -d '{"index": {"\_source" : > > {"compress" : true} }}' > > curl -XPUT localhost:9200/order/\_settings -d '{"index": {"\_source" : > > {"compress\_threshold" : 10000} }} > > > > I really appreciate any help on this. > > > > Please give me some guidelines how to scale large data. > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/compression-flag-is-not-wrking-tp3927247p3927247.html](http://elasticsearch-users.115913.n3.nabble.com/compression-flag-is-not-wrking-tp3927247p3927247.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:31am UTC](https://discuss.elastic.co/t/compression-flag-is-not-wrking/7428/4 "2017-07-06T03:31:09Z")