Integration Ingest pipeline not executed when logstash ouptut is activated for Agent Policy (Fleet)

[mehdi-lamrani]

8.7 here

For some obscure reason,
when I add a pipeline to an integration via Custom configurations (lower red rectangle in first screen below), it is not triggered when the policy integration output is set to logstash (second screen below).

I need to switch it back to Elasticsearch for the ingest pipeline to kick in.
Looks like I cant have both Logstash output AND ingest pipeline set this way, for some reason.

Which is a shame because they offer different degrees of granularity (integration level pipelines, policy level pipelines)

PS : I am quasi sure I made both work at the same time at some point but I can't seem to make it work again.

Thanks for your help

Screenshot 2023-05-09 at 13.14.12

[mehdi-lamrani]

Do I need to do this ? (Not sure if it makes sense...)

Use ingest pipelines for parsing | Logstash Reference [8.7] | Elastic

[mehdi-lamrani]

Up ? Anyone ?

Is it possible to use both Logstash output and Ingest pipeline at the same time or not ?

There is no mention that one break the other in the documentation

[Anton_H (Anton)]

If you ouput from the agent to logstash then events come from logstash to elasticsearch, I think it was possible to pass pipelines but your case proves otherwise.
I think it would be best to have the pipeline linked to your index template. That way the source of the events is irrelevant and your events should be processed by the pipeline when they are sent to the index linked to the index template.

edit: are you sure you have configured pipeline => "%{[@metadata][pipeline]}" in your logstash ouput?
If this is what your mentioning, then yes, you need to do that.

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "ELASTIC_HOST"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create" 
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "ELASTIC_HOST""
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create"
      user => "elastic"
      password => "secret"
    }
  }
}