I ran 'optimize' on old Logstash indices with 'max_num_segments=1' to improve search performance and maybe reclaim some disk space, however, I noticed that 'optimize' actually caused -most- indices to use more disk space; the cluster size went from 4TB to 4.4TB!
The cluster is running ES 1.7.2 on 5 nodes with 5 shards and 0 replicas, I'm wondering if this is expected behavior?
If you have high cardinality in some of the values in fields and use doc values on them it could be why.
I see; you're right about the high cardinality (logs including Windows events) and doc values which are enabled.
I guess my confusion is to why the substantial increase in size, I would expect it to at least stay the same.
How much of a difference was it?
About 400GB overall; I logged the size before and after optimization for each index, here is a sample of it:
2015/11/06 01:24:17 - logstash-abc-2015.08.29: optimizing 24411.48MB...
2015/11/06 01:42:58 - logstash-abc-2015.08.29: optimized 27676.86MB.
2015/11/06 01:43:03 - logstash-abc-2015.08.30: optimizing 27769.69MB...
2015/11/06 02:04:43 - logstash-abc-2015.08.30: optimized 31528.48MB.
2015/11/06 02:04:44 - logstash-abc-2015.08.31: optimizing 43472.18MB...
2015/11/06 02:42:55 - logstash-abc-2015.08.31: optimized 49044.40MB.
2015/11/06 02:42:55 - logstash-abc-2015.09.01: optimizing 45268.35MB...
2015/11/06 03:19:20 - logstash-abc-2015.09.01: optimized 51135.11MB.
2015/11/06 03:19:20 - logstash-abc-2015.09.02: optimizing 44285.74MB...
2015/11/06 03:55:15 - logstash-abc-2015.09.02: optimized 49844.92MB.
2015/11/06 03:55:18 - logstash-abc-2015.09.03: optimizing 45416.78MB...
2015/11/06 04:37:07 - logstash-abc-2015.09.03: optimized 51484.99MB.
2015/11/06 04:37:10 - logstash-abc-2015.09.04: optimizing 26249.26MB...
2015/11/06 05:04:23 - logstash-abc-2015.09.04: optimized 29161.84MB.
2015/11/06 05:04:26 - logstash-abc-2015.09.05: optimizing 25739.71MB...
2015/11/06 05:30:56 - logstash-abc-2015.09.05: optimized 29082.64MB.
2015/11/06 05:30:56 - logstash-abc-2015.09.06: optimizing 28353.10MB...
2015/11/06 05:51:38 - logstash-abc-2015.09.06: optimized 32434.70MB.
2015/11/06 05:51:40 - logstash-abc-2015.09.07: optimizing 43804.41MB...
2015/11/06 06:30:17 - logstash-abc-2015.09.07: optimized 49626.81MB.
2015/11/06 06:30:18 - logstash-abc-2015.09.08: optimizing 44581.38MB...
2015/11/06 07:07:18 - logstash-abc-2015.09.08: optimized 50740.98MB.
2015/11/06 07:07:19 - logstash-abc-2015.09.09: optimizing 43394.43MB...
2015/11/06 07:49:25 - logstash-abc-2015.09.09: optimized 49135.58MB.
2015/11/06 07:49:26 - logstash-abc-2015.09.10: optimizing 42023.62MB...
2015/11/06 08:24:41 - logstash-abc-2015.09.10: optimized 47714.74MB.
2015/11/06 08:24:41 - logstash-abc-2015.09.11: optimizing 24654.35MB...
2015/11/06 08:43:19 - logstash-abc-2015.09.11: optimized 27832.03MB.
High cardinality fields and doc values can be a reason like Mark said. For instance if all your values are unique and you have two segments that have 1M unique values each, then the merged segment will have 2M unique values, which requires one more bit per document for addressing. These isn't really anything that can be done about it, this is just the way things are designed.
Another potential reason are sparse fields with doc values. For efficiency reasons, elasticsearch needs to reserve space for documents that don't have a value. Imagine you have 2 segments, segment 1 has values for field foo but segment 2 does not. So field foo does not require any disk space on segment 2, but as soon as you merge those segments, elasticsearch will suddenly need to reserve some space for all documents of segment 2 even though they don't have a value for 'foo'. This is something that we hope to improve soon in the extreme cases (when less than 1% of documents have a value for a given field). You can see https://issues.apache.org/jira/browse/LUCENE-6863 for more information.
How are you measuring the index size? Are you running ES on Windows (which refuses to delete still-open files)?
Be sure to flush after optimizing, otherwise the old segments may still be referenced (by either the last commit point, or the last refreshed reader, or both) and consuming disk space even though they are effectively "ghosts".
@mikemccand ES is running on CentOS, I'm calling 'stats' and using 'size_in_bytes' to get the index size. I didn't refresh because it defaults to true when running optimize as per the documentation.
@jpountz Thanks, this makes more sense now.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.