# Does Elasticsearch automatically compress data?

**URL:** <https://discuss.elastic.co/t/does-elasticsearch-automatically-compress-data/93161>\
**Category:** Elasticsearch\
**Created:** [July 14, 2017, 10:05am UTC](https://discuss.elastic.co/t/does-elasticsearch-automatically-compress-data/93161 "2017-07-14T10:05:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adam\_Szabo](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@Adam\_Szabo](https://discuss.elastic.co/u/Adam_Szabo)\
**Post date:** [July 14, 2017, 10:05am UTC](https://discuss.elastic.co/t/does-elasticsearch-automatically-compress-data/93161/1 "2017-07-14T10:05:27Z")

</div>

2 days ago I imported 144 MB of log to see how much space is needed for them in Elasticsearch.  
The result was:

```
/var/lib/elasticsearch# du -ch nodes/ 
...
974M nodes/0
974M nodes/
974M total

```

Today I checked that directory again, and now it has a much lower size:

```
/var/lib/elasticsearch# du -ch nodes/
...
316M nodes/0
316M nodes/
316M total

```

The biggest difference is:

```
< 129M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/3/translog
< 190M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/3
---
> 8.0K nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/3/translog
> 62M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/3
108,110c108,110
< 136M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/1/translog
< 197M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/1
< 61M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/2/index
---
> 8.0K nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/1/translog
> 63M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/1
> 62M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/2/index
112,114c112,114
< 136M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/2/translog
< 197M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/2
< 62M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/0/index
---
> 8.0K nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/2/translog
> 62M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/2
> 63M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/0/index
116,117c116,117
< 129M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/0/translog
< 190M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/0
---
> 8.0K nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/0/translog
> 63M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/0
119c119
< 62M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/4/index
---
> 63M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/4/index
121,127c121,127
< 136M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/4/translog
< 197M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/4
< 970M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A
< 974M nodes/0/indices
< 974M nodes/0
< 974M nodes/
< 974M total
\ No newline at end of file
---
> 8.0K nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/4/translog
> 63M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A/4
> 310M nodes/0/indices/nEHLUvAWQhi6IMQtqiTM9A
> 316M nodes/0/indices
> 316M nodes/0
> 316M nodes/
> 316M total
\ No newline at end of file

```

Can someone explain please what is happening?  
I'd like to see some exact numbers so that I could order the production server. 🙂

I'm using Elasticsearch 5.3.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [July 14, 2017, 3:25pm UTC](https://discuss.elastic.co/t/does-elasticsearch-automatically-compress-data/93161/2 "2017-07-14T15:25:04Z")

</div>

Does Elasticsearch automatically compress? Yes. The default compression is LZ4, but you _can_ use DEFLATE, which is higher compression at the cost of more CPU to compress further, and a slightly slower indexing rate.

With regards to space, the first big thing I notice is that the translogs seem to be clearing themselves out—just like they're supposed to. The translog (transaction log) keeps a running record of things being indexed. Those entries are purged out of the translog after they have been verified to have been added to Elasticsearch. This process is transparent to the user, and runs continuously, so long as new data is being added to Elasticsearch.

For your purposes, consider the disk space needed as something in the vein of FIFO space for things coming in, that will empty as soon as it can. You may need to plan for that space to balloon and shrink as noted above.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 11, 2017, 3:25pm UTC](https://discuss.elastic.co/t/does-elasticsearch-automatically-compress-data/93161/3 "2017-08-11T15:25:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.