Data stream index template is resetting

jsheely · July 17, 2024, 10:50pm

One of my data stream index templates settings and mappings keeps resetting to the default values and I can't figure out why

The primary setting I've changed is total_fields limit from 2,000 to 10,000 but for some reason the settings revert back to 2,000 after a while. Typically during a rollover but not always

It is also resetting my custom mappings.

What is changing this back and how do I keep it from happening?

Thank you in advance

nfeekery · July 22, 2024, 1:30pm

From Elastic Search to Elasticsearch

jsheely · August 4, 2024, 11:28pm

So for anyone else coming here from the C# world.

The problem is the Elastic Serilog Sink when Bootstrap is set to anything but None for DataStreams

The SDK does a lookup to see if a data stream template exists and unless it returns a status code 200 it will attempt to PUT a new template. Even if one already exists. Which this can happen if there is a network hiccup for whatever reason when it does the lookup.

There should be some Polly retry logic or additional checks in the event of a 502 error or something similar but at this time there is not.

References:

github.com

elastic/ecs-dotnet/blob/f10a76169f93be8d9a69b56c2547873b7eefbdef/src/Elastic.Ingest.Elasticsearch.CommonSchema/EcsDataStreamChannel.cs#L78


      
          		}
          
          		/// <summary>
          		/// Bootstrap the target data stream. Will register the appropriate index and component templates
          		/// </summary>
          		/// <param name="bootstrapMethod">Either None (no bootstrapping), Silent (quiet exit), Failure (throw exceptions)</param>
          		/// <param name="ilmPolicy">Registers a component template that ensures the template is managed by this ilm policy</param>
          		/// <param name="ctx"></param>
          		public override async Task<bool> BootstrapElasticsearchAsync(BootstrapMethod bootstrapMethod, string? ilmPolicy = null, CancellationToken ctx = default)
          		{
          			if (bootstrapMethod == BootstrapMethod.None) return true;
          
          			var name = TemplateName;
          			var match = TemplateWildcard;
          			if (await IndexTemplateExistsAsync(name, ctx).ConfigureAwait(false)) return false;
          
          			foreach (var kv in IndexComponents.Components)
          			{
          				if (!await PutComponentTemplateAsync(bootstrapMethod, kv.Key, kv.Value, ctx).ConfigureAwait(false))
          					return false;
          			}

github.com

elastic/elastic-ingest-dotnet/blob/fbcd631c6873b1a662b4984723326421bf20325f/src/Elastic.Ingest.Elasticsearch/ElasticsearchChannelBase.Bootstrap.cs#L90


      
          			return false;
          
          		return true;
          	}
          
          	/// <summary></summary>
          	protected bool IndexTemplateExists(string name)
          	{
          		var templateExists = Options.Transport.Request<HeadIndexTemplateResponse>(HttpMethod.HEAD, $"_index_template/{name}");
          		var statusCode = templateExists.ApiCallDetails.HttpStatusCode;
          		return statusCode is 200;
          	}
          
          	/// <summary></summary>
          	protected async Task<bool> IndexTemplateExistsAsync(string name, CancellationToken ctx = default)
          	{
          		var templateExists = await Options.Transport.RequestAsync<HeadIndexTemplateResponse>
          				(HttpMethod.HEAD, $"_index_template/{name}", cancellationToken: ctx)
          			.ConfigureAwait(false);
          		var statusCode = templateExists.ApiCallDetails.HttpStatusCode;
          		return statusCode is 200;

system · September 1, 2024, 11:28pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Default index template for streams Elasticsearch datastreams	1	419	April 27, 2021
Mapping modifications in index template breaks search queries Elasticsearch	18	461	July 6, 2017
Overriding Builtin Datastream Index / Component Templates Elasticsearch elastic-stack-monitoring	16	1245	February 10, 2021
Error when using template generated by ECS Tooling (1.7 and master) for datastream Elasticsearch ecs-elastic-common-schema	8	3172	March 11, 2021
How to Handle Index Template Changes with Data Streams Elasticsearch	1	284	December 2, 2020

Data stream index template is resetting

Related topics