Pipeline with id [] does not exist

Elastic Stack Elasticsearch
1

There are few references to this problem already here but none provide resolution, I am getting the below error on 7.6.2 and i do see the pipeline when i run the GET API call

 {
      "index" : "xxxxxxxxxx-2020.45-reindex",
      "type" : "_doc",
      "id" : "AXWZCt_d8Su-BwbMoSsb",
      "cause" : {
        "type" : "illegal_argument_exception",
        "reason" : "pipeline with id [mypipe] does not exist"
      },
      "status" : 400

Any suggestions what can be tried to fix this ?

2

There is no pipeline named mypipe. That's the only thing we can tell from what you shared.

What is the output of:

GET /_ingest/pipeline/mypipe

Please read this about how to format.

3

Hi David,

Appreciate the response, the output is shared below

{
  "mypipe" : {
    "description" : "ELK Migration",
    "processors" : [
      {
        "rename" : {
          "field" : "host",
          "target_field" : "host.name"
        }
      }
    ]
  }
}
4

Could you explain exactly what you did?

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script is something anyone can copy and paste in Kibana dev console, click on the run button to reproduce your use case. It will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

5

Thank you for the prompt responses, i used the below two snippets to create a pipeline and then force a reindex (old to new cluster) using the above pipeline.

I have removed sensitive info

curl -uxxxxxx:xxxxxx -X PUT "#########:9200/_ingest/pipeline/mypipe?pretty" -H 'Content-Type: application/json' -d'
{
  "description" : "ELK Migration",
  "processors" : [
    {
      "rename": {
        "field": "host",
        "target_field": "host.name"
      }
    }
  ]
}
'


curl -uxxxxxx:xxxxxxx -X POST "#########:9200/_reindex?pretty=true" -H 'Content-Type:application/json' -d'
{
  "source": {
    "remote": {"host": "http://#########:9200", "username": "elastic", "password": "xxxxxx"},
    "index": "inf_cmk_log-2020.45",
  "dest":
    "index": "inf_cmk_log-2020.45-reindex",
    "pipeline": "mypipe",
	"op_type": "create",
  }
}
'
6

On which cluster did you run the 2 commands?
Could you replace the xxxxx values with cluster1 cluster2 so it's more obvious?

7

Sorry About that !

curl -uelastic:xxxxxx -X PUT "clu01:9200/_ingest/pipeline/mypipe?pretty" -H 'Content-Type: application/json' -d'{
  "description" : "ELK Migration",
  "processors" : [
    {
      "rename": {
        "field": "host",
        "target_field": "host.name"
      }
    }
  ]
}
'

curl   -XPOST   -uelastic:xxxxxx  "clu01:9200/_reindex?wait_for_completion=true&pretty=true" -H 'Content-Type:application/json' -d'
{
  "source": {
    "remote": {"host": "http://clu02:9200", "username": "elastic", "password": "xxxxxx"},
    "index": "inf_cmk_log-2020.45",
  "dest":
    "index": "inf_cmk_log-2020.45-reindex",
    "pipeline": "mypipe",
	"op_type": "create",
  }
}
'
8

What is the full output of:

GET /_ingest/pipeline/

Do not modify the content please.

9 
Hope this helps

{"xpack_monitoring_6":{"description":"This pipeline upgrades documents from the older version of the Monitoring API to the newer version (7) by fixing breaking changes in those older documents before they are indexed from the older version (6).","version":7000199,"processors":[{"script":{"source":"ctx._type = null"}},{"gsub":{"field":"_index","pattern":"(.monitoring-\\w+-)6(-.+)","replacement":"$17$2"}}]},"xpack_monitoring_7":{"description":"This is a placeholder pipeline for Monitoring API version 7 so that future versions may fix breaking changes.","version":7000199,"processors":[]},"mypipe":{"description":"ELK Migration","processors":[{"rename":{"field":"host","target_field":"host.name"}}]}}[root@clu01 ~]#
10

Which elastic version are you using on both clusters?

Could you run on clu01:

DELETE test 
PUT test/_doc/1?pipeline=mypipe
{ "foo": "bar" }

?

11

There is a difference, the old cluster is on 5.6.3 and new one is 7.6.2

12

will test the delete and get back to you..

13

Really not sure how however this is resolved by a server reboot. I was able to use the pipeline, @dadoonet Appreciate your expertise !

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.