In version 8.17.3, some default pipelines are automatically created after being deleted using DELETE _ingest/pipeline/*

Elastic Stack Elasticsearch
1
  1. Install elasticsearch 8.17.3 and find that there are many default pipelines
  2. Run the DELETE _ingest/pipeline/* and _ingest/pipeline/pipeline_id command. The command output is deleted successfully
  3. But with get, the discovery is recreated, and the log is as follows:

image
image661×255 33.6 KB

  1. Add configuration in elasticsaerch.yml
    xpack.apm_data.enabled: false
    tracing.apm.enabled: false
    The pipelines that are closed first by apm can be deleted and are not automatically generated
  2. The following pipeline will be automatically created. Why? What are the roles of these pipelines? Does deletion have any effect? How can I prevent these pipelines from being generated automatically
    pipeline: logs@json-pipeline
    logs-default-pipeline
    logs@json-message
    ent-search-generic-ingestion
    search-default-ingestion
    behavioral_analytics-events-final_pipeline
    logs@default-pipeline
2

Looking forward to the official response, I also have the same question :grinning_face:

3

Hello and welcome,

What is the reason for deleting these? They are used by multiple tools in Kibana and other parts of the stack and not having it may break some things.

If I'm not wrong they are used by the built-in templates which are controlled by the stack.templates.enabled setting, so you would need to set it to false.

Keep in mind that this is not recommended, from the documentation you have this note.

It is not recommended to disable the built-in stack templates, as some functionality of Elasticsearch or Kibana will not work correctly when disabled. Features like log and metric collection, as well as Kibana reporting, may malfunction without the built-in stack templates. Stack templates should only be disabled temporarily, if necessary, to resolve upgrade issues, then re-enabled after any issues have been resolved.

5

@leandrojmp Thank you for your reply. We will get all pipelines, including default and custom ones. The default pipelines will cause trouble to users, so we want to delete these default pipelines. We also tried to add stack.templates.enabled to false. Since you said it is not recommended to do so, how can I avoid the confusion caused by the default pipeline

6

Can I suggest another approach? Well, 2.

  1. Open a support case, if using a paid license(s). I mean the license is relatively costly, so make use of it when you hit an issue that has business impact for you..
  2. Open a bug report, documenting the specifics of all the confusion and the issues/difficulties that creates for your users. Your user community are typically creating pipelines, so are pretty skilled and knowledgable already?
7

The technical reason is that there is a piece of code that is listening for any change to the cluster state, and when that happens it makes sure to recreate the system pipelines if they don't exist: elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/template/IndexTemplateRegistry.java at main · elastic/elasticsearch · GitHub. From what I can tell, there is no way around this -- that clusterChanged method is called on all ClusterStateListener objects fairly frequently.