Upgrading from 7.x to 8.x with soft deletes disabled

Elastic Stack Elasticsearch
1

Hi,

We use Elasticsearch as a lightweight index rather than a primary data store. The actual documents live elsewhere, and ES is only used to retrieve matching document IDs.

Our current Elasticsearch 7.x setup includes:

  • _source disabled

  • Soft deletes disabled

  • doc_values disabled on one field, which is a large array of strings within each document

As we plan an upgrade to Elasticsearch 8.x, we understand that soft deletes are now mandatory and that parts of this configuration are no longer supported.

Given that constraint, are there any known workarounds or alternative configurations in 8.x that still allow for a storage-efficient, ID-only index?

2

Can you expand on this a bit? Are you really searching via Elasticsearch? Or is it effectively being used as a kind of KV store, and any “searches” are maybe better described as filtering?

redis is a better fit for lots of use cases. It’s just not clear if this might be one of them.

Also, why are you updating at all? Is there some feature of 8.x that you need ? If so, which ?

3

Hi Kevin,

I’m not using Elasticsearch as a KV store. Each document has a field that contains a list of strings, and I want to query for documents that contain those strings using AND/OR logic. For example:

(string1 OR string2) AND string3

As for the upgrade, the latest version of Elasticsearch 7.x depends on an end-of-life version of Ubuntu (20.04.6), which is the main driver for the upgrade. I’m not aware of any plans to update Elasticsearch 7.x with a new ubuntu version.

4

Thanks for answer.

I realize IT is littered with not universally well-defined terminology, but whatever you have, which is being a little bit drip-fed, is not what I would understand as a "ID-only index"? That is how you described it in the first post on the thread. Others might disagree here.

There will be no more releases to 7.x. But the latest 7.x releases support both Ubuntu 22.04 and RHEL 9. The latter of which will be supported for some time yet, for different values of "supported".

You asked for "workarounds or alternative configurations in 8.x". For the latter at least, I'd want to know more details than you have currently shared to be able to answer?

Can you share the mapping for the index/indices?

5

Thanks Kevin.

The latest 7.x Docker images are built on Ubuntu 20.04. Are there any plans to move to a newer, supported Ubuntu version?

The mapping for our index is shown below. Each document contains a tags field, which is a large array of strings. We need to query for documents that match specific combinations of tags using AND/OR logic.

We are only interested in retrieving the contentId for each matching document and do not need the _source, since the full documents are stored in a separate datastore.

Additionally, the index is rebuilt daily and is immutable after creation, with no updates or deletions.

{
	"index": {
		"mappings": {
			"dynamic": "strict",
			"_source": {
				"enabled": false
			},
			"properties": {
				"tags": {
					"type": "keyword",
					"doc_values": false
				},
				"contentid": {
					"type": "keyword"
				}
			}
		}
	}
}
6

I don't work for Elastic, so don't know. But I doubt it. There was a thread on here recently when someone was building docker images him/herself .... I guess you could do the same ... personally, I would not go there.

And your answer really takes me back to my first thought. Your tags are all keywords, so without getting into semantics I'd say your "searches” are maybe better described as logical filters. "(tag1 OR tag2) AND tag3". And honestly, I think redis is a decent alternative if my understanding is correct. I presume "large" in "large array of strings" is "a lot", often thousands or tens/hundreds of thousands?

BUT, if " the index is rebuilt daily and is immutable after creation, with no updates or deletions", why are you worrying about soft_deletes? What am I missing ?

EDIT: I re-read this and is your concern only at the daily "index building" phase, i.e. the index is constructed via a sequence of updates to same IDs, maybe adding more and more tags to same document, before it reaches its final, immutable state? And without the ability to disable soft_deletes you are concerned the disk usage will increase significantly during the index construction phase? if so, can you add some numbers? And did you test it on 8.x at all, validate the concern would be a real issue?

7

I was bored so did little test, all on 7.latest.

I created 2 indices, tags mapped to keyword field, no docvalues, no _source, and only difference was that soft deletes is enabled on one and disabled on the other. All changes are applied identically to docs in both indices. Each index starts with 100k documents with some random strings in tags.

index names are tagtest-soft-deletes-disabled & tagtest-soft-deletes-enabled

I then loop endlessly, just updating a random doc at each pass, adding another value to the tags array for that specific ID. So there is some slow natural growth in the index size, as more tags appear in some IDs. I used an English wordlist for the tags, just choose words randomly.

Every 20k updates I do a forcemerge.

Output is below. Clearly OOTB the disk space for tagtest-soft-deletes-enabled is significantly more, but once a forcemerge is run the reported disk usage is not so different between the indices with enabled and disabled soft deletes.

I didn't test anything wrt query performance.

Obviously this is just s small synthetic test, but maybe it suggests that while building your daily index, using forcemerge to manage your disk footprint might be good enough, both on 7.,x and 8.x should be no different in this respect.

Obviously all depends on your actual numbers.

[2026-01-29 17:18:04] Performed       0 updates. Disk usage: tagtest-soft-deletes-disabled    7.49 MB. tagtest-soft-deletes-enabled   24.29 MB.
[2026-01-29 17:18:25] Performed    2000 updates. Disk usage: tagtest-soft-deletes-disabled    7.50 MB. tagtest-soft-deletes-enabled   24.37 MB.
[2026-01-29 17:18:46] Performed    4000 updates. Disk usage: tagtest-soft-deletes-disabled    7.52 MB. tagtest-soft-deletes-enabled   24.64 MB.
[2026-01-29 17:19:07] Performed    6000 updates. Disk usage: tagtest-soft-deletes-disabled    7.53 MB. tagtest-soft-deletes-enabled   24.86 MB.
[2026-01-29 17:19:43] Performed    8000 updates. Disk usage: tagtest-soft-deletes-disabled    7.54 MB. tagtest-soft-deletes-enabled   25.07 MB.
[2026-01-29 17:20:34] Performed   10000 updates. Disk usage: tagtest-soft-deletes-disabled    7.55 MB. tagtest-soft-deletes-enabled   25.24 MB.
[2026-01-29 17:21:24] Performed   12000 updates. Disk usage: tagtest-soft-deletes-disabled    7.56 MB. tagtest-soft-deletes-enabled   25.40 MB.
[2026-01-29 17:22:15] Performed   14000 updates. Disk usage: tagtest-soft-deletes-disabled    7.57 MB. tagtest-soft-deletes-enabled   25.62 MB.
[2026-01-29 17:23:07] Performed   16000 updates. Disk usage: tagtest-soft-deletes-disabled    7.58 MB. tagtest-soft-deletes-enabled   25.83 MB.
[2026-01-29 17:23:58] Performed   18000 updates. Disk usage: tagtest-soft-deletes-disabled    7.59 MB. tagtest-soft-deletes-enabled   25.99 MB.
[2026-01-29 17:24:53] Performed   20000 updates. Disk usage: tagtest-soft-deletes-disabled    7.60 MB. tagtest-soft-deletes-enabled   26.21 MB.
[2026-01-29 17:25:14] Performed   22000 updates. Disk usage: tagtest-soft-deletes-disabled    7.62 MB. tagtest-soft-deletes-enabled    7.88 MB.
[2026-01-29 17:25:35] Performed   24000 updates. Disk usage: tagtest-soft-deletes-disabled    7.63 MB. tagtest-soft-deletes-enabled    8.10 MB.
[2026-01-29 17:25:57] Performed   26000 updates. Disk usage: tagtest-soft-deletes-disabled    7.64 MB. tagtest-soft-deletes-enabled    8.26 MB.
[2026-01-29 17:26:24] Performed   28000 updates. Disk usage: tagtest-soft-deletes-disabled    7.65 MB. tagtest-soft-deletes-enabled    8.42 MB.
[2026-01-29 17:27:14] Performed   30000 updates. Disk usage: tagtest-soft-deletes-disabled    7.66 MB. tagtest-soft-deletes-enabled    8.64 MB.
[2026-01-29 17:28:04] Performed   32000 updates. Disk usage: tagtest-soft-deletes-disabled    7.68 MB. tagtest-soft-deletes-enabled    8.85 MB.
[2026-01-29 17:28:55] Performed   34000 updates. Disk usage: tagtest-soft-deletes-disabled    7.68 MB. tagtest-soft-deletes-enabled    9.02 MB.
[2026-01-29 17:29:45] Performed   36000 updates. Disk usage: tagtest-soft-deletes-disabled    7.70 MB. tagtest-soft-deletes-enabled    9.23 MB.
[2026-01-29 17:30:34] Performed   38000 updates. Disk usage: tagtest-soft-deletes-disabled    7.70 MB. tagtest-soft-deletes-enabled    9.45 MB.
[2026-01-29 17:31:30] Performed   40000 updates. Disk usage: tagtest-soft-deletes-disabled    7.73 MB. tagtest-soft-deletes-enabled    7.86 MB.
[2026-01-29 17:31:51] Performed   42000 updates. Disk usage: tagtest-soft-deletes-disabled    7.73 MB. tagtest-soft-deletes-enabled    7.96 MB.
[2026-01-29 17:32:12] Performed   44000 updates. Disk usage: tagtest-soft-deletes-disabled    7.74 MB. tagtest-soft-deletes-enabled    8.23 MB.
[2026-01-29 17:32:33] Performed   46000 updates. Disk usage: tagtest-soft-deletes-disabled    7.75 MB. tagtest-soft-deletes-enabled    8.39 MB.
[2026-01-29 17:33:00] Performed   48000 updates. Disk usage: tagtest-soft-deletes-disabled    7.77 MB. tagtest-soft-deletes-enabled    8.61 MB.
[2026-01-29 17:33:50] Performed   50000 updates. Disk usage: tagtest-soft-deletes-disabled    7.77 MB. tagtest-soft-deletes-enabled    8.77 MB.
[2026-01-29 17:34:42] Performed   52000 updates. Disk usage: tagtest-soft-deletes-disabled    7.79 MB. tagtest-soft-deletes-enabled    8.99 MB.
[2026-01-29 17:35:32] Performed   54000 updates. Disk usage: tagtest-soft-deletes-disabled    7.79 MB. tagtest-soft-deletes-enabled    9.21 MB.
[2026-01-29 17:36:23] Performed   56000 updates. Disk usage: tagtest-soft-deletes-disabled    7.81 MB. tagtest-soft-deletes-enabled    9.37 MB.
[2026-01-29 17:37:12] Performed   58000 updates. Disk usage: tagtest-soft-deletes-disabled    7.82 MB. tagtest-soft-deletes-enabled    9.59 MB.
[2026-01-29 17:38:09] Performed   60000 updates. Disk usage: tagtest-soft-deletes-disabled    7.84 MB. tagtest-soft-deletes-enabled    7.90 MB.
[2026-01-29 17:38:30] Performed   62000 updates. Disk usage: tagtest-soft-deletes-disabled    7.84 MB. tagtest-soft-deletes-enabled    8.05 MB.
[2026-01-29 17:38:51] Performed   64000 updates. Disk usage: tagtest-soft-deletes-disabled    7.85 MB. tagtest-soft-deletes-enabled    8.21 MB.
[2026-01-29 17:39:12] Performed   66000 updates. Disk usage: tagtest-soft-deletes-disabled    7.86 MB. tagtest-soft-deletes-enabled    8.43 MB.
[2026-01-29 17:39:39] Performed   68000 updates. Disk usage: tagtest-soft-deletes-disabled    7.87 MB. tagtest-soft-deletes-enabled    8.65 MB.
[2026-01-29 17:40:30] Performed   70000 updates. Disk usage: tagtest-soft-deletes-disabled    7.88 MB. tagtest-soft-deletes-enabled    8.81 MB.
[2026-01-29 17:41:22] Performed   72000 updates. Disk usage: tagtest-soft-deletes-disabled    7.89 MB. tagtest-soft-deletes-enabled    9.02 MB.
[2026-01-29 17:42:14] Performed   74000 updates. Disk usage: tagtest-soft-deletes-disabled    7.90 MB. tagtest-soft-deletes-enabled    9.24 MB.
[2026-01-29 17:43:05] Performed   76000 updates. Disk usage: tagtest-soft-deletes-disabled    7.91 MB. tagtest-soft-deletes-enabled    9.40 MB.
[2026-01-29 17:43:56] Performed   78000 updates. Disk usage: tagtest-soft-deletes-disabled    7.92 MB. tagtest-soft-deletes-enabled    9.56 MB.
[2026-01-29 17:44:51] Performed   80000 updates. Disk usage: tagtest-soft-deletes-disabled    7.92 MB. tagtest-soft-deletes-enabled    9.84 MB.
[2026-01-29 17:45:12] Performed   82000 updates. Disk usage: tagtest-soft-deletes-disabled    7.93 MB. tagtest-soft-deletes-enabled    8.18 MB.
...
[2026-01-29 19:10:15] Performed  314000 updates. Disk usage: tagtest-soft-deletes-disabled    8.70 MB. tagtest-soft-deletes-enabled   10.09 MB.
[2026-01-29 19:11:05] Performed  316000 updates. Disk usage: tagtest-soft-deletes-disabled    8.70 MB. tagtest-soft-deletes-enabled   10.31 MB.
[2026-01-29 19:11:57] Performed  318000 updates. Disk usage: tagtest-soft-deletes-disabled    8.72 MB. tagtest-soft-deletes-enabled   10.52 MB.
[2026-01-29 19:12:53] Performed  320000 updates. Disk usage: tagtest-soft-deletes-disabled    8.73 MB. tagtest-soft-deletes-enabled   10.69 MB.
[2026-01-29 19:13:14] Performed  322000 updates. Disk usage: tagtest-soft-deletes-disabled    8.68 MB. tagtest-soft-deletes-enabled    8.88 MB.
[2026-01-29 19:13:36] Performed  324000 updates. Disk usage: tagtest-soft-deletes-disabled    8.69 MB. tagtest-soft-deletes-enabled    9.09 MB.
[2026-01-29 19:13:57] Performed  326000 updates. Disk usage: tagtest-soft-deletes-disabled    8.70 MB. tagtest-soft-deletes-enabled    9.31 MB.
[2026-01-29 19:14:47] Performed  328000 updates. Disk usage: tagtest-soft-deletes-disabled    8.71 MB. tagtest-soft-deletes-enabled    9.54 MB.
[2026-01-29 19:15:37] Performed  330000 updates. Disk usage: tagtest-soft-deletes-disabled    8.72 MB. tagtest-soft-deletes-enabled    9.74 MB.
[2026-01-29 19:16:26] Performed  332000 updates. Disk usage: tagtest-soft-deletes-disabled    8.73 MB. tagtest-soft-deletes-enabled    9.91 MB.
...
[2026-01-29 22:54:28] Performed  902000 updates. Disk usage: tagtest-soft-deletes-disabled    9.83 MB. tagtest-soft-deletes-enabled   10.18 MB.
[2026-01-29 22:55:20] Performed  904000 updates. Disk usage: tagtest-soft-deletes-disabled    9.84 MB. tagtest-soft-deletes-enabled   10.40 MB.
[2026-01-29 22:56:09] Performed  906000 updates. Disk usage: tagtest-soft-deletes-disabled    9.85 MB. tagtest-soft-deletes-enabled   10.67 MB.
[2026-01-29 22:56:58] Performed  908000 updates. Disk usage: tagtest-soft-deletes-disabled    9.86 MB. tagtest-soft-deletes-enabled   10.94 MB.
[2026-01-29 22:57:48] Performed  910000 updates. Disk usage: tagtest-soft-deletes-disabled    9.87 MB. tagtest-soft-deletes-enabled   11.16 MB.
[2026-01-29 22:58:40] Performed  912000 updates. Disk usage: tagtest-soft-deletes-disabled    9.89 MB. tagtest-soft-deletes-enabled   11.42 MB.
[2026-01-29 22:59:30] Performed  914000 updates. Disk usage: tagtest-soft-deletes-disabled    9.89 MB. tagtest-soft-deletes-enabled   11.65 MB.
[2026-01-29 23:00:20] Performed  916000 updates. Disk usage: tagtest-soft-deletes-disabled   10.84 MB. tagtest-soft-deletes-enabled   12.91 MB.
[2026-01-29 23:01:11] Performed  918000 updates. Disk usage: tagtest-soft-deletes-disabled   10.87 MB. tagtest-soft-deletes-enabled   13.19 MB.
[2026-01-29 23:02:07] Performed  920000 updates. Disk usage: tagtest-soft-deletes-disabled    9.89 MB. tagtest-soft-deletes-enabled   10.01 MB.
[2026-01-29 23:02:57] Performed  922000 updates. Disk usage: tagtest-soft-deletes-disabled    9.89 MB. tagtest-soft-deletes-enabled   10.18 MB.
2 Likes