Issue while running a pipeline

Elastic Stack Elasticsearch
1

Hi Team,

I'm trying to run elser and ner pipelines through the reindexing and I'm having the following error :
pipeline with id [elser_pipeline_peopleagg] could not be loaded, caused by [org.elasticsearch.ElasticsearchParseException: Error updating pipeline with id [elser_pipeline_peopleagg]; org.elasticsearch.ElasticsearchStatusException: Max number of inference processors reached, total inference processors [50]. Adjust the setting [xpack.ml.max_inference_processors]: [50] if a greater number is desired.]

Just for reference, reindexing script :
{
"source": {
"remote": {
"host": "https",
"username": "d",
"password": "d"
},
"index": "peopleagg_09282023",
"size": 100,
"query": {
"bool": {
"must": {
"bool": {
"should": [
{
"exists": {
"field": "aboutme"
}
},
{
"exists": {
"field": "resumecontent"
}
}
]
}
},
"must_not": {
"bool": {
"should": [
{
"term": {
"aboutme.keyword": ""
}
},
{
"term": {
"resumecontent.keyword": ""
}
}
]
}
}
}
},
"_source": [
"aboutme",
"accountname",
"badgedescription",
"badgename",
"email",
"pdifunctionsglobaltext",
"pdicitydesc",
"pdicountrydesc",
"pdiempattributesjobtitledesc",
"pdiempattributespreffirstname",
"pdiempattributespreflastname",
"pdiemployeesmfid",
"pdiprimaryindustryglobaltext",
"pdisecondaryindustryglobaltext",
"resumecontent",
"role",
"URL",
"pdilevelsglobaltext",
"pdiorganizationsorganizationdesc"
]
},
"dest": {
"index": "peopleaggml_110202023",
"pipeline": "elser_pipeline_peopleagg"
}
}
I have been stuck on this. Can someone please help me on resolving this?

Thanks,
Manasa.

2

Hello @Manasa4,

The total number of inference type processors allowed across all ingest pipelines was reached. This value defaults to 50 to prevent excessive resource usage. If you need more, and to resolve this issue, you can increase the limit of inference processors.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.