Elastic Agent to Logstash - how to access pipeline metadata

Elastic Stack Logstash
1

We’re setting up an Elastic Agent → Logstash → Elasticsearch architecture to leverage Logstash’s buffering capabilities, which ingest nodes don’t provide.

Our goal is to route all Elastic Agent traffic through Logstash.

We’ve noticed that data sent from the Elastic Agent through Logstash appears to be automatically routed to the correct Ingest Pipeline in Elasticsearch (even without modifying the data in Logstash).

Now we’re trying to understand how to access the pipeline metadata within Logstash. We checked the @metadata fields but couldn’t find any [@metadata][pipeline] field or similar.
Since the data clearly reaches the appropriate ingest pipeline, this information must exist somewhere. We just can’t see where.

2

When writing to Data Streams with the elasticsearch output, Logstash uses the data_stream.* fields from the event to know which data stream it needs to send the request.

The events from Elastic Agent will have the fields data_stream.type, data_stream.dataset and data_stream.namespace, so Logstash uses this information when making the request to Elasticsearch.

The information about the pipeline is configured in the data stream settings, logstash just direct the request to the correct data stream.