Hi, I am using Elastic.Serilog.Sinks to stream some data directly to elasticsearch a very simple mechanism
.WriteTo.Logger(lc =>
{
lc.Filter.ByIncludingOnly(le => le.Properties.ContainsKey("ecs.version"))
.WriteTo.Elasticsearch(
new[] { new Uri("http://localhost:9200") },
opts =>
{
opts.DataStream = new DataStreamName("logs", entryAssembly.GetName().Name!.ToLowerInvariant());
opts.BootstrapMethod = BootstrapMethod.Failure;
opts.ConfigureChannel = channelOpts =>
{
channelOpts.BufferOptions = new BufferOptions { ExportMaxConcurrency = 10 };
};
},
_ =>
{
// No authentication
});
})
thing is it created addtitional fields i doint need and want like agent.type and so on. I thought i can create blacklist pipeline something like
private static async Task IngestPipeline()
{
var pipelineDefinition = new
{
description = "Removes ECS fields we don't want",
processors = new object[]
{
new
{
remove = new
{
field = new[]
{
"agent.type",
"agent.version",
"data_stream.type"
},
ignore_missing = true
}
}
}
};
string json = JsonConvert.SerializeObject(pipelineDefinition);
using var httpClient = new HttpClient();
var requestContent = new StringContent(json, Encoding.UTF8, "application/json");
string pipelineUrl = "http://localhost:9200/_ingest/pipeline/remove_ecs_fields";
using HttpResponseMessage response = await httpClient.PutAsync(pipelineUrl, requestContent);
response.EnsureSuccessStatusCode();
}
but is there some event trigger in ElasticsearchSinkOptions I can use to execute it?
for example:
opts.Pipeline = "remove_ecs_fields";