LogsDB Without Synthetic _Source: Storage Savings and Upgrade Risks?

Elastic Stack Elasticsearch
1

I am currently on Elasticsearch version 8.16 and considering using the LogsDB index mode for its benefits, particularly the improved querying speed and reduced storage footprint with the new sorting feature (up to 20%, based on the Elastic blog). However, I understand that starting with version 8.17, the synthetic _source feature is only available with an Enterprise license.

Since I do not want to risk a sudden increase in log storage size after enabling the logsdb and later upgrading to 8.17 (if we do not have a valid Enterprise license), I want to know:

  1. Is it possible to enable LogsDB mode to take advantage of sorting-based storage savings and faster queries but disable synthetic _source? (Cause I didn't find any setting for this)
  2. Is there any recommended approach to avoid potential issues when upgrading from 8.16 to 8.17 while using LogsDB, particularly without an Enterprise license?

Thanks in advance for any input

2

Yes, you can use logsdb without synthetic_source, I don't think that a specific setting exists, it will use synthetic_source if you have an enterprise license and the normal _source if you don't.

At least this is what is mentioned here in the documentation.

If you don’t have the required subscription, logsdb mode uses the original _source field.

1 Like
3

@leandrojmp Thanks for the fast reply. This is something that I am concerned about. Using the logsdb index mode, I will benefit from the synthetic_source feature in my Elastic cluster (version 8.16) if there is no setting to disable this, which, together with smart sorting, provides around 50-60% storage savings.

However, if I upgrade to version 8.17 or later in future, the synthetic_source feature will be ignored as I don't have the required license. This will result in a big reversion of the storage savings, significantly impacting cluster storage occupancy.

For instance, we are ingesting around 300 GB of logs per day, so a 20-30% reversion would be significant

4

Hi @Behnam.R

Great questions, but I think we are all perhaps missing something.

LogsDB in 8.16 is in Technical Preview mode... we would never suggest that you use this in production, especially at the scale you are working with. Technical Preview means that the capability is subject to change..... including breaking changes which it has / is effectivly with the 8.17 GA / licensing change.

The documentation literally says this

Logs data streams and the logsdb index mode are in tech preview and may be changed or removed in the future. Don’t use logs data streams or logsdb index mode in production.

So we / I would suggest that you not enable logsdb in production before you upgrade to 8.17 in the first place.

Then, all the required logic will be in place.

2 Likes
5

@leandrojmp @Behnam.R
To close it out a bit...
Turns out there is a default value and setting for non synthetic stored etc.. and yup it needs to be better documented

6

thanks for the update, as you suggested earlier I have upgraded the cluster to 8.17.0, but now I am also aware of the extra settings :slight_smile: