Circuit_breaking_exception when creating an ingest pipeline

[val (Val Crettaz)]

I'm trying to create an ingest pipeline with a simple remove processor that contains a lot of field names to remove from the document (> 100) prior to indexing. There is no script processor in the pipeline and I have left the default script.max_compilations_rate setting (i.e. 75/5m)

The pipeline looks like this:

PUT _ingest/pipeline/my-pipeline
{
  "description": "Test pipeline",
  "processors": [
    {
      "remove": {
        "field": [
          "field1",
          "field2",
          "field3",
          "field4",
          "field5",
          "field6",
          "field7",
          "field8",
          "field9",
          "field10",
          "field11",
          "field12",
          "field13",
          "field14",
          "field15",
          "field16",
          "field17",
          "field18",
          "field19",
          "field20",
          "field21",
          "field22",
          "field23",
          "field24",
          "field25",
          "field26",
          "field27",
          "field28",
          "field29",
          "field30",
          "field31",
          "field32",
          "field33",
          "field34",
          "field35",
          "field36",
          "field37",
          "field38",
          "field39",
          "field40",
          "field41",
          "field42",
          "field43",
          "field44",
          "field45",
          "field46",
          "field47",
          "field48",
          "field49",
          "field50",
          "field51",
          "field52",
          "field53",
          "field54",
          "field55",
          "field56",
          "field57",
          "field58",
          "field59",
          "field60",
          "field61",
          "field62",
          "field63",
          "field64",
          "field65",
          "field66",
          "field67",
          "field68",
          "field69",
          "field70",
          "field71",
          "field72",
          "field73",
          "field74",
          "field75",
          "field76",
          "field77",
          "field78",
          "field79",
          "field80",
          "field81",
          "field82",
          "field83",
          "field84",
          "field85",
          "field86",
          "field87",
          "field88",
          "field89",
          "field90",
          "field91",
          "field92",
          "field93",
          "field94",
          "field95",
          "field96",
          "field97",
          "field98",
          "field99",
          "field100",
          "field101",
          "field102",
          "field103",
          "field104",
          "field105"
        ],
        "ignore_failure": true
      }
    }
  ],
  "on_failure": [
    {
      "append": {
        "field": "meta.errors",
        "value": "{{ _ingest.on_failure_message }}, {{ _ingest.on_failure_processor_type }}, {{ _ingest.on_failure_processor_tag }}"
      }
    }
  ]
}

This command results in the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "circuit_breaking_exception",
        "reason": "[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting",
        "bytes_wanted": 0,
        "bytes_limit": 0
      }
    ],
    "type": "general_script_exception",
    "reason": "Failed to compile inline script [field15] using lang [mustache]",
    "caused_by": {
      "type": "circuit_breaking_exception",
      "reason": "[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting",
      "bytes_wanted": 0,
      "bytes_limit": 0
    },
    "header": {
      "processor_type": "remove",
      "property_name": "field"
    }
  },
  "status": 500
}

The only reason I think I'm getting a script error here is because the processors are getting translated to painless scripts underneath. Am I correct? The only way I can circumvent this is to increase the max_compilations_rate setting.

[val (Val Crettaz)]

Ok, I've kind of figured it out by looking at the source code for RemoveProcessor where we can clearly see that scripting is involved when configuring the processor. From there on, the error "makes sense", even though it was not clear at the beginning that ingest processors were implemented using scripting.

Using Mustache scripts for removing fields feels a bit like using a canon to shoot a fly, but there must be some context there, so I'll simply stick with the max_compilations_rate setting