DataStream creates hidden indicas

Elastic Stack Elasticsearch
1

I use Elastic.Serilog.Sinks lib to send logs to Elastic with Serilog. This code sends logs to ES, but the created indexes are hidden. Why?

How can I set the index unhidden?
How can I set the index format?

  public static void AddSerilog(this WebApplicationBuilder builder, IConfiguration configuration,
        string environment,
        string projectName)
    {
        var uri = configuration["Elasticsearch:Url"]!;

        Log.Logger = new LoggerConfiguration()
            .MinimumLevel.Debug()
            .Enrich.FromLogContext()
            .Enrich.WithElasticApmCorrelationInfo()
            .Enrich.WithProperty("Project", projectName)
            .Enrich.WithProperty("Environment", environment)
            .Enrich.WithProperty("Coder", "Jeyhun Rahimov")
            .WriteTo.Console(new EcsTextFormatter())
            .WriteTo.Elasticsearch(new[] { new Uri(uri) }, opts =>
            {
                opts.DataStream = new DataStreamName("logs", projectName.ToLower(), environment.ToLower());
                opts.TextFormatting = new EcsTextFormatterConfiguration();
                opts.BootstrapMethod = BootstrapMethod.Failure;
                opts.ConfigureChannel = channelOpts =>
                {
                    channelOpts.BufferOptions = new BufferOptions
                    {
                        ExportMaxConcurrency = 10
                    };
                };
            })
            .CreateLogger();

        builder.Logging.ClearProviders();
        builder.Host.UseSerilog(Log.Logger, true);
    }

it creates hidden index like:
.ds-logs-auth-development-2024.07.18-000001

2

Hello and welcome,

This is how datastreams works as described in the documentation.

A data stream consists of one or more hidden, auto-generated backing indices.

I don't think you can, why you would want to do that?

Not clear what you mean with index format, can you provide more context?

3

Thanks for the quick response.
I just wanted to make them unhidden. if it is not possible, it is ok. Thanks

By index format, I mean changing the name structure of the index:
.ds-logs-auth-development-2024.07.18-000001
remove the ".ds" part etc.

And last question:
I have different spaces in Elastic: Demo1, Demo2.
How can I send logs to a specific space, ie to Demo1?

4

That's also not possible, check the linked documentation about data streams, it explain how they work.

The .ds-* prefix is part of every data streams.

Spaces are a Kibana concept, you do not send logs do spaces, you send your logs to Elasticsearch and then in Kibana you configure your Data Views and select in which spaces they are visible.

In your Kibana go to Stack Management > Data Views you will be able to change in which space each data view will appear.

1 Like