Logstash is not creating ingest pipelines

[Mocem (marco)]

Hi All,

For the past few days I'm struggling with loading the pipelines from filebeat. Basically nothing is happening. It works directly to Elastic but not when logstash is in between. Once I create the ingest pipeline manually it starts to log without a problem.

The weird thing is I now have 2 test setups one with SSL and one without. Both behave the same. For an unknown reason there is very little logging and the documentation is pretty clear how to load pipelines. The must be something I'm missing I was hoping someone could point me in the right direction.

Logstash keeps logging the following messages.

"status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"pipeline with id [filebeat-7.12.1-system-auth-pipeline] does not exist"}}}}

logstash config

input {
  beats {
    port => 5400
    host => "0.0.0.0"
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => ["http://10.0.15.80:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}"
    }
  } else {
    elasticsearch {
      hosts => ["http://10.0.15.80:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

Thanks in advance

Marco

[warkolm (Mark Walkom)]

Pipelines need to be loaded directly into Elasticsearch from the Beat, not via Logstash. See Load ingest pipelines | Filebeat Reference [8.11] | Elastic;

If you’re sending events to Logstash, or plan to use Beats central management, you need to load the ingest pipelines manually.

[Mocem (marco)]

Thank you for the quick reply.