Failed to start transform

Hi
I'm trying to start a transform but keep getting errors and turns the cluster's state into red (sooooo... frustrating )
last message on Kinana is "Created transform."

Any help will be appreciated (removing the painless script doesnt help)

PUT _transform/daily
{
  
  "source": {
    "index": [
      "metrics-*"
    ],
    "query": 
    {
      "bool": {
        "must": [
          {
            "range": {
              "datetime": {
                "gte": "now-3d"
              }
            }
          },
          {
            "exists": {
              "field": "value"
            }
          }
        ]
      }
    }
  },
  "settings":{
    "docs_per_second":500
  },
  "pivot": {
    "group_by": {
      "clientname": {
        "terms": {
          "script": {
            "source": "if (doc.containsKey('clientname.keyword')) { return doc['clientname.keyword'].value} else { return doc['clientname'].value}"
          }
        }
      },
      "servername": {
        "terms": {
          "field": "servername"
        }
      },
      "category": {
        "terms": {
          "field": "category"
        }
      },
      "counter": {
        "terms": {
          "field": "counter"
        }
      },
      "instance": {
        "terms": {
          "field": "instance"
        }
      },
      "datetime": {
        "date_histogram": {
          "field": "datetime",
          "calendar_interval": "1d"
        }
      }
    },
    "aggregations": {
      "clientid": {
        "max": {
          "field": "clientid"
        }
      },
      "avg_value": {
        "avg": {
          "field": "value"
        }
      },
      "percentiles_value": {
        "percentiles": {
          "field": "value",
          "percents": [
            4,
            11,
            90,
            97
          ]
        }
      },
      "min_value": {
        "min": {
          "field": "value"
        }
      },
      "max_value": {
        "max": {
          "field": "value"
        }
      }
    }
  },
  "dest": {
    "index": "daily"
  },
  "frequency": "1h", 
  "sync": {
  
    "time": {
      "field": "datetime",
      "delay": "1h"
    }
  }
}

Why not use support instead of posting here? You are a cloud customer so you have support access...

Hi @richcollier
I'm a standard customer in the cloud, and usually they only send links to documentation but i'll try.

Is there anything in the Elasticsearch logs?

@warkolm

I think I've found the problem.
1 of 85 indices, that the transform was referring to, had a wrong mapping- text instead of keyword.

I still don't understand why it kills the cluster, and causing 200+ unassigned shards.

from the logs: (nothing more meaningful )

[instance-0000000072] Not starting transform [daily], because not all primary shards are active for the following indices [.transform-internal-005,.transform-internal-004,.transform-internal-003]

ok there is something in the logs

The bottom log is when the transform starts
then elastic update replicas to 0 and to 1
then it cannot start the transform as the cluster became red state

HI @liorg2

The indices have "auto_expand_replicas" : "0-1", so that fact they are jumping from zero and 1 replicas implies that an additional node has appeared. It seems unlikely to me that you added additional nodes at the exact same time as you were starting transforms.

I cannot comment on the cause of this, but it feels unlikely that a PUT transform action could achieve this. Your screenshot shows there are no transforms running, so this statement isn't moving any data around yet. However if you've seen earlier logging and it is repeatable then it could be a trigger and requires investigation.

The issue you are seeing implies an instability in your cloud environment and could be related to its configuration. I would recommend you seek help from our cloud support specialists. They have more experience and tooling relevant for this issue.

Best wishes
Sophie

its actually the POST _transform/.../start
thanks @sophie_chang very appreciated

Apparently this is a bug in elasticsearch that will be fixed in the next minor version :face_with_head_bandage:
IMHO I think that transform became a very critical resource in elastic, and bugs in it should be fixed immediately